Description of problem: Unfortunately I do not yet have a manually repeatable test scenario for this bug, but our nightly test automation runs have been hitting this Runtime Error regularly against Candlepin master since July 11-ish, 2013. On the command line from the subscription-manager client.... 201308020231:22.850 - FINE: ssh root.redhat.com ls -1 -v /etc/pki/entitlement/*.pem | grep -v key.pem 201308020231:22.919 - FINE: Stdout: /etc/pki/entitlement/4098574580346795424.pem /etc/pki/entitlement/5626592194288091665.pem 201308020231:22.920 - FINE: Stderr: 201308020231:22.921 - FINE: ExitCode: 0 201308020231:22.921 - FINE: ssh root.redhat.com subscription-manager unregister 201308020231:24.126 - FINE: Stdout: Runtime Error No row with the given identifier exists: [org.candlepin.model.ProvidedProduct#8a90f81a403bb40601403db8ca435015] at org.hibernate.UnresolvableObjectException.throwIfNull:65 201308020231:24.126 - FINE: Stderr: 201308020231:24.126 - FINE: ExitCode: 255 Corresponding catalina.out from candlepin... Aug 02 02:31:17 [http-8443-10] INFO org.candlepin.model.PoolCurator - Omitting pool due to failed rule check: 8a90f81a403bb40601403bb57df90786 Aug 02 02:31:17 [http-8443-10] INFO org.candlepin.model.PoolCurator - Errors: [rulefailed.consumer.type.mismatch] Aug 02 02:31:17 [http-8443-10] INFO org.candlepin.model.PoolCurator - Omitting pool due to failed rule check: 8a90f81a403bb40601403bb579160597 Aug 02 02:31:17 [http-8443-10] INFO org.candlepin.model.PoolCurator - Warnings: [rulewarning.architecture.mismatch] Aug 02 02:31:24 [http-8443-10] ERROR org.candlepin.exceptions.mappers.CandlepinExceptionMapper - Runtime Error No row with the given identifier exists: [org.candlepin.model.ProvidedProduct#8a90f81a403bb40601403db8ca435015] at org.hibernate.UnresolvableObjectException.throwIfNull:65 org.hibernate.UnresolvableObjectException: No row with the given identifier exists: [org.candlepin.model.ProvidedProduct#8a90f81a403bb40601403db8ca435015] at org.hibernate.UnresolvableObjectException.throwIfNull(UnresolvableObjectException.java:65) at org.hibernate.event.def.DefaultRefreshEventListener.onRefresh(DefaultRefreshEventListener.java:148) at org.hibernate.impl.SessionImpl.fireRefresh(SessionImpl.java:939) at org.hibernate.impl.SessionImpl.refresh(SessionImpl.java:922) at org.hibernate.engine.CascadingAction$3.cascade(CascadingAction.java:192) at org.hibernate.engine.Cascade.cascadeToOne(Cascade.java:292) at org.hibernate.engine.Cascade.cascadeAssociation(Cascade.java:240) at org.hibernate.engine.Cascade.cascadeProperty(Cascade.java:193) at org.hibernate.engine.Cascade.cascadeCollectionElements(Cascade.java:320) at org.hibernate.engine.Cascade.cascadeCollection(Cascade.java:266) at org.hibernate.engine.Cascade.cascadeAssociation(Cascade.java:243) at org.hibernate.engine.Cascade.cascadeProperty(Cascade.java:193) at org.hibernate.engine.Cascade.cascade(Cascade.java:154) at org.hibernate.event.def.DefaultRefreshEventListener.onRefresh(DefaultRefreshEventListener.java:121) at org.hibernate.event.def.DefaultRefreshEventListener.onRefresh(DefaultRefreshEventListener.java:62) at org.hibernate.impl.SessionImpl.fireRefresh(SessionImpl.java:930) at org.hibernate.impl.SessionImpl.refresh(SessionImpl.java:918) at org.candlepin.model.PoolCurator.lockAndLoad(PoolCurator.java:455) at org.candlepin.controller.CandlepinPoolManager.removeEntitlement(CandlepinPoolManager.java:767) at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) at org.candlepin.controller.CandlepinPoolManager.revokeAllEntitlements(CandlepinPoolManager.java:836) at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) at org.candlepin.resource.ConsumerResource.deleteConsumer(ConsumerResource.java:1119) at org.candlepin.auth.interceptor.SecurityInterceptor.invoke(SecurityInterceptor.java:127) at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:66) at sun.reflect.GeneratedMethodAccessor420.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) at org.candlepin.servlet.filter.ContentTypeHackFilter.doFilter(ContentTypeHackFilter.java:58) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at org.candlepin.servlet.filter.logging.LoggingFilter.doFilter(LoggingFilter.java:60) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at org.candlepin.servlet.filter.CandlepinPersistFilter.doFilter(CandlepinPersistFilter.java:48) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at org.candlepin.servlet.filter.CandlepinScopeFilter.doFilter(CandlepinScopeFilter.java:57) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:636) Version-Release number of selected component (if applicable): [root@jsefler-f14-5candlepin candlepin]# git branch * master [root@jsefler-f14-5candlepin candlepin]# git show-ref | head -1 85940a01293f29bd927e6e1c81fa2f584286d8ac refs/heads/master
Created attachment 782083 [details] catalina.out containing six occurances of "Runtime Error No row with the given identifier exists" Attaching catalina.out which contains these six occurances of this Runtime Error during a nightly test automation run... [root@jsefler-f14-5candlepin tomcat6]# grep "No row" catalina.out Aug 02 02:31:24 [http-8443-10] ERROR org.candlepin.exceptions.mappers.CandlepinExceptionMapper - Runtime Error No row with the given identifier exists: [org.candlepin.model.ProvidedProduct#8a90f81a403bb40601403db8ca435015] at org.hibernate.UnresolvableObjectException.throwIfNull:65 org.hibernate.UnresolvableObjectException: No row with the given identifier exists: [org.candlepin.model.ProvidedProduct#8a90f81a403bb40601403db8ca435015] Aug 02 02:32:01 [http-8443-10] ERROR org.candlepin.exceptions.mappers.CandlepinExceptionMapper - Runtime Error No row with the given identifier exists: [org.candlepin.model.DerivedProvidedProduct#8a90f81a403bb40601403db941c55026] at org.hibernate.UnresolvableObjectException.throwIfNull:65 org.hibernate.UnresolvableObjectException: No row with the given identifier exists: [org.candlepin.model.DerivedProvidedProduct#8a90f81a403bb40601403db941c55026] Aug 02 02:32:04 [http-8443-10] ERROR org.candlepin.exceptions.mappers.CandlepinExceptionMapper - Runtime Error No row with the given identifier exists: [org.candlepin.model.DerivedProvidedProduct#8a90f81a403bb40601403db941c55026] at org.hibernate.UnresolvableObjectException.throwIfNull:65 org.hibernate.UnresolvableObjectException: No row with the given identifier exists: [org.candlepin.model.DerivedProvidedProduct#8a90f81a403bb40601403db941c55026] Aug 02 02:32:07 [http-8443-10] ERROR org.candlepin.exceptions.mappers.CandlepinExceptionMapper - Runtime Error No row with the given identifier exists: [org.candlepin.model.DerivedProvidedProduct#8a90f81a403bb40601403db941c55026] at org.hibernate.UnresolvableObjectException.throwIfNull:65 org.hibernate.UnresolvableObjectException: No row with the given identifier exists: [org.candlepin.model.DerivedProvidedProduct#8a90f81a403bb40601403db941c55026] Aug 02 02:32:10 [http-8443-10] ERROR org.candlepin.exceptions.mappers.CandlepinExceptionMapper - Runtime Error No row with the given identifier exists: [org.candlepin.model.DerivedProvidedProduct#8a90f81a403bb40601403db941c55026] at org.hibernate.UnresolvableObjectException.throwIfNull:65 org.hibernate.UnresolvableObjectException: No row with the given identifier exists: [org.candlepin.model.DerivedProvidedProduct#8a90f81a403bb40601403db941c55026] Aug 02 10:50:06 [http-8443-5] ERROR org.candlepin.exceptions.mappers.CandlepinExceptionMapper - Runtime Error No row with the given identifier exists: [org.candlepin.model.DerivedProvidedProduct#8a90f81a403bb40601403db941c55026] at org.hibernate.UnresolvableObjectException.throwIfNull:65 org.hibernate.UnresolvableObjectException: No row with the given identifier exists: [org.candlepin.model.DerivedProvidedProduct#8a90f81a403bb40601403db941c55026]
The error always appears to be in CandlepinPoolManager.removeEntitlement. The call stack can be one of two paths: DELETE /consumers/uuid/entitlements (revoke all entitlements) -> ConsumerResource.unbindAll -> CandlepinPoolManager.removeEntitlement DELETE/consumers/uuid (delete consumer) -> ConsumerResource.deleteConsumer -> CandlepinPoolManager.revokeAllEntitlements -> CandlepinPoolManager.removeEntitlement John: I'm assuming you must repeatedly call these two functions. Could you provide some info on how they're used? Do you always for example do a "delete all entitlements", then "delete the consumer" in some kind of teardown? (only one of those is actually necessary) Do you do much asynchronously where jobs are fired off simultaneously or do you always block and wait for a call to complete before moving on? Similar question for owner cleanup/deletion? Is there any cleanup of owners or anything else that might delete pools, that could be running around the same time as the consumer cleanup?
I have not seen this error in awhile. Let's close this issue for now.