| Summary: | Race condition on pools/owner deletion | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Community] Candlepin | Reporter: | Ivan Necas <inecas> | ||||
| Component: | candlepin | Assignee: | Jesus M. Rodriguez <jesusr> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Katello QA List <katello-qa-list> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 0.5 | CC: | bkearney, jesusr, wpoteat | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2015-03-19 12:09:59 UTC | Type: | --- | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Attachments: |
|
||||||
Sometimes it fails on deleting owner with this exception: Feb 09 11:28:05 [http-8443-1] INFO org.candlepin.resource.OwnerResource - Cleaning up owner: Owner [name = test, key = test, id = 4028fa813561a769013561a7d2d80003] Feb 09 11:28:05 [http-8443-1] INFO org.candlepin.resource.OwnerResource - Deleting pool: EntitlementPool [id = 4028fa813561a769013561a7d7040013, owner = 4028fa813561a769013561a7d2d80003, products = NATURE - [org.candlepin.model.ProvidedProduct@52751a9b], sub = 4028fa813561a769013561a7d60a000a, quantity = 30, expires = 2025-05-29 02:00:00.0] Feb 09 11:28:05 [http-8443-1] ERROR org.hibernate.jdbc.AbstractBatcher - Exception executing batch: org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1 at org.hibernate.jdbc.Expectations$BasicExpectation.checkBatched(Expectations.java:85) at org.hibernate.jdbc.Expectations$BasicExpectation.verifyOutcome(Expectations.java:70) at org.hibernate.jdbc.BatchingBatcher.checkRowCounts(BatchingBatcher.java:90) at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70) at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268) at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:114) at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:109) at org.hibernate.jdbc.AbstractBatcher.prepareBatchStatement(AbstractBatcher.java:244) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:2497) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:2715) at org.hibernate.action.EntityDeleteAction.execute(EntityDeleteAction.java:97) at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:172) at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321) at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:50) at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1028) at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:366) at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:137) at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:54) at com.wideplay.warp.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:88) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.google.inject.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44) at org.candlepin.model.PoolCurator$$EnhancerByGuice$$1de1355a.delete(<generated>) at org.candlepin.resource.OwnerResource.cleanupAndDelete(OwnerResource.java:317) at org.candlepin.resource.OwnerResource.deleteOwner(OwnerResource.java:276) at org.candlepin.resource.OwnerResource$$EnhancerByGuice$$546512a2.CGLIB$deleteOwner$8(<generated>) at org.candlepin.resource.OwnerResource$$EnhancerByGuice$$546512a2$$FastClassByGuice$$29f9aae1.invoke(<generated>) at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at org.candlepin.auth.interceptor.SecurityInterceptor.invoke(SecurityInterceptor.java:118) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.google.inject.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44) at org.candlepin.resource.OwnerResource$$EnhancerByGuice$$546512a2.deleteOwner(<generated>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140) at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:255) at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:220) at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:209) at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:519) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:496) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:216) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:141) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:63) at org.candlepin.servlet.filter.ContentTypeHackFilter.doFilter(ContentTypeHackFilter.java:61) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59) at org.candlepin.servlet.filter.logging.LoggingFilter.doFilter(LoggingFilter.java:60) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59) at com.wideplay.warp.persist.PersistenceFilter$3.run(PersistenceFilter.java:141) at com.wideplay.warp.persist.internal.Lifecycles.failEarlyAndLeaveNoOneBehind(Lifecycles.java:29) at com.wideplay.warp.persist.PersistenceFilter.doFilter(PersistenceFilter.java:155) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:122) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:110) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:679) Feb 09 11:28:05 [http-8443-1] ERROR org.hibernate.event.def.AbstractFlushingEventListener - Could not synchronize database state with session org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1 at org.hibernate.jdbc.Expectations$BasicExpectation.checkBatched(Expectations.java:85) at org.hibernate.jdbc.Expectations$BasicExpectation.verifyOutcome(Expectations.java:70) at org.hibernate.jdbc.BatchingBatcher.checkRowCounts(BatchingBatcher.java:90) at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70) at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268) at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:114) at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:109) at org.hibernate.jdbc.AbstractBatcher.prepareBatchStatement(AbstractBatcher.java:244) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:2497) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:2715) at org.hibernate.action.EntityDeleteAction.execute(EntityDeleteAction.java:97) at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:172) at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321) at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:50) at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1028) at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:366) at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:137) at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:54) at com.wideplay.warp.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:88) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.google.inject.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44) at org.candlepin.model.PoolCurator$$EnhancerByGuice$$1de1355a.delete(<generated>) at org.candlepin.resource.OwnerResource.cleanupAndDelete(OwnerResource.java:317) at org.candlepin.resource.OwnerResource.deleteOwner(OwnerResource.java:276) at org.candlepin.resource.OwnerResource$$EnhancerByGuice$$546512a2.CGLIB$deleteOwner$8(<generated>) at org.candlepin.resource.OwnerResource$$EnhancerByGuice$$546512a2$$FastClassByGuice$$29f9aae1.invoke(<generated>) at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at org.candlepin.auth.interceptor.SecurityInterceptor.invoke(SecurityInterceptor.java:118) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.google.inject.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44) at org.candlepin.resource.OwnerResource$$EnhancerByGuice$$546512a2.deleteOwner(<generated>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140) at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:255) at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:220) at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:209) at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:519) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:496) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:216) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:141) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:63) at org.candlepin.servlet.filter.ContentTypeHackFilter.doFilter(ContentTypeHackFilter.java:61) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59) at org.candlepin.servlet.filter.logging.LoggingFilter.doFilter(LoggingFilter.java:60) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59) at com.wideplay.warp.persist.PersistenceFilter$3.run(PersistenceFilter.java:141) at com.wideplay.warp.persist.internal.Lifecycles.failEarlyAndLeaveNoOneBehind(Lifecycles.java:29) at com.wideplay.warp.persist.PersistenceFilter.doFilter(PersistenceFilter.java:155) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:122) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:110) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:679) Feb 09 11:28:05 [QuartzScheduler_Worker-9] ERROR org.hibernate.jdbc.AbstractBatcher - Exception executing batch: org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1 at org.hibernate.jdbc.Expectations$BasicExpectation.checkBatched(Expectations.java:85) at org.hibernate.jdbc.Expectations$BasicExpectation.verifyOutcome(Expectations.java:70) at org.hibernate.jdbc.BatchingBatcher.checkRowCounts(BatchingBatcher.java:90) at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70) at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268) at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:114) at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:109) at org.hibernate.jdbc.AbstractBatcher.prepareBatchStatement(AbstractBatcher.java:244) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:2497) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:2715) at org.hibernate.action.EntityDeleteAction.execute(EntityDeleteAction.java:97) at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:172) at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321) at org.hibernate.event.def.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:64) at org.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:997) at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1590) at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:306) at org.candlepin.model.PoolCurator.entitlementsIn(PoolCurator.java:289) at org.candlepin.model.PoolCurator$$EnhancerByGuice$$1de1355a.CGLIB$entitlementsIn$10(<generated>) at org.candlepin.model.PoolCurator$$EnhancerByGuice$$1de1355a$$FastClassByGuice$$9e2a3eaa.invoke(<generated>) at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.wideplay.warp.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.google.inject.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44) at org.candlepin.model.PoolCurator$$EnhancerByGuice$$1de1355a.entitlementsIn(<generated>) at org.candlepin.controller.CandlepinPoolManager.deletePool(CandlepinPoolManager.java:774) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7.CGLIB$deletePool$9(<generated>) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7$$FastClassByGuice$$633c9893.invoke(<generated>) at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.wideplay.warp.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.google.inject.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7.deletePool(<generated>) at org.candlepin.controller.CandlepinPoolManager.refreshPools(CandlepinPoolManager.java:186) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7.CGLIB$refreshPools$2(<generated>) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7$$FastClassByGuice$$633c9893.invoke(<generated>) at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.ProxyFactory$1.intercept(ProxyFactory.java:53) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7.refreshPools(<generated>) at org.candlepin.pinsetter.tasks.RefreshPoolsJob.execute(RefreshPoolsJob.java:66) at org.candlepin.pinsetter.tasks.RefreshPoolsJob$$EnhancerByGuice$$fbd1fef5.CGLIB$execute$0(<generated>) at org.candlepin.pinsetter.tasks.RefreshPoolsJob$$EnhancerByGuice$$fbd1fef5$$FastClassByGuice$$4ebc4602.invoke(<generated>) at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.wideplay.warp.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:68) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.google.inject.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44) at org.candlepin.pinsetter.tasks.RefreshPoolsJob$$EnhancerByGuice$$fbd1fef5.execute(<generated>) at org.candlepin.pinsetter.core.TransactionalPinsetterJob.execute(TransactionalPinsetterJob.java:50) at org.quartz.core.JobRunShell.run(JobRunShell.java:199) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546) Feb 09 11:28:05 [QuartzScheduler_Worker-9] ERROR org.hibernate.event.def.AbstractFlushingEventListener - Could not synchronize database state with session org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1 at org.hibernate.jdbc.Expectations$BasicExpectation.checkBatched(Expectations.java:85) at org.hibernate.jdbc.Expectations$BasicExpectation.verifyOutcome(Expectations.java:70) at org.hibernate.jdbc.BatchingBatcher.checkRowCounts(BatchingBatcher.java:90) at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70) at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268) at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:114) at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:109) at org.hibernate.jdbc.AbstractBatcher.prepareBatchStatement(AbstractBatcher.java:244) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:2497) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:2715) at org.hibernate.action.EntityDeleteAction.execute(EntityDeleteAction.java:97) at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:172) at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321) at org.hibernate.event.def.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:64) at org.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:997) at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1590) at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:306) at org.candlepin.model.PoolCurator.entitlementsIn(PoolCurator.java:289) at org.candlepin.model.PoolCurator$$EnhancerByGuice$$1de1355a.CGLIB$entitlementsIn$10(<generated>) at org.candlepin.model.PoolCurator$$EnhancerByGuice$$1de1355a$$FastClassByGuice$$9e2a3eaa.invoke(<generated>) at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.wideplay.warp.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.google.inject.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44) at org.candlepin.model.PoolCurator$$EnhancerByGuice$$1de1355a.entitlementsIn(<generated>) at org.candlepin.controller.CandlepinPoolManager.deletePool(CandlepinPoolManager.java:774) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7.CGLIB$deletePool$9(<generated>) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7$$FastClassByGuice$$633c9893.invoke(<generated>) at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.wideplay.warp.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.google.inject.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7.deletePool(<generated>) at org.candlepin.controller.CandlepinPoolManager.refreshPools(CandlepinPoolManager.java:186) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7.CGLIB$refreshPools$2(<generated>) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7$$FastClassByGuice$$633c9893.invoke(<generated>) at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.ProxyFactory$1.intercept(ProxyFactory.java:53) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7.refreshPools(<generated>) at org.candlepin.pinsetter.tasks.RefreshPoolsJob.execute(RefreshPoolsJob.java:66) at org.candlepin.pinsetter.tasks.RefreshPoolsJob$$EnhancerByGuice$$fbd1fef5.CGLIB$execute$0(<generated>) at org.candlepin.pinsetter.tasks.RefreshPoolsJob$$EnhancerByGuice$$fbd1fef5$$FastClassByGuice$$4ebc4602.invoke(<generated>) at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.wideplay.warp.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:68) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.google.inject.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44) at org.candlepin.pinsetter.tasks.RefreshPoolsJob$$EnhancerByGuice$$fbd1fef5.execute(<generated>) at org.candlepin.pinsetter.core.TransactionalPinsetterJob.execute(TransactionalPinsetterJob.java:50) at org.quartz.core.JobRunShell.run(JobRunShell.java:199) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546) Feb 09 11:28:05 [QuartzScheduler_Worker-9] ERROR org.quartz.core.JobRunShell - Job async group.refresh_pools_94cec021-4dd2-45ff-9979-1554381bab9e threw an unhandled Exception: org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1 at org.hibernate.jdbc.Expectations$BasicExpectation.checkBatched(Expectations.java:85) at org.hibernate.jdbc.Expectations$BasicExpectation.verifyOutcome(Expectations.java:70) at org.hibernate.jdbc.BatchingBatcher.checkRowCounts(BatchingBatcher.java:90) at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70) at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268) at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:114) at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:109) at org.hibernate.jdbc.AbstractBatcher.prepareBatchStatement(AbstractBatcher.java:244) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:2497) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:2715) at org.hibernate.action.EntityDeleteAction.execute(EntityDeleteAction.java:97) at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:172) at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321) at org.hibernate.event.def.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:64) at org.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:997) at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1590) at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:306) at org.candlepin.model.PoolCurator.entitlementsIn(PoolCurator.java:289) at org.candlepin.model.PoolCurator$$EnhancerByGuice$$1de1355a.CGLIB$entitlementsIn$10(<generated>) at org.candlepin.model.PoolCurator$$EnhancerByGuice$$1de1355a$$FastClassByGuice$$9e2a3eaa.invoke(<generated>) at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.wideplay.warp.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.google.inject.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44) at org.candlepin.model.PoolCurator$$EnhancerByGuice$$1de1355a.entitlementsIn(<generated>) at org.candlepin.controller.CandlepinPoolManager.deletePool(CandlepinPoolManager.java:774) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7.CGLIB$deletePool$9(<generated>) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7$$FastClassByGuice$$633c9893.invoke(<generated>) at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.wideplay.warp.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.google.inject.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7.deletePool(<generated>) at org.candlepin.controller.CandlepinPoolManager.refreshPools(CandlepinPoolManager.java:186) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7.CGLIB$refreshPools$2(<generated>) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7$$FastClassByGuice$$633c9893.invoke(<generated>) at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.ProxyFactory$1.intercept(ProxyFactory.java:53) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7.refreshPools(<generated>) at org.candlepin.pinsetter.tasks.RefreshPoolsJob.execute(RefreshPoolsJob.java:66) at org.candlepin.pinsetter.tasks.RefreshPoolsJob$$EnhancerByGuice$$fbd1fef5.CGLIB$execute$0(<generated>) at org.candlepin.pinsetter.tasks.RefreshPoolsJob$$EnhancerByGuice$$fbd1fef5$$FastClassByGuice$$4ebc4602.invoke(<generated>) at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.wideplay.warp.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:68) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.google.inject.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44) at org.candlepin.pinsetter.tasks.RefreshPoolsJob$$EnhancerByGuice$$fbd1fef5.execute(<generated>) at org.candlepin.pinsetter.core.TransactionalPinsetterJob.execute(TransactionalPinsetterJob.java:50) at org.quartz.core.JobRunShell.run(JobRunShell.java:199) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546) Feb 09 11:28:05 [QuartzScheduler_Worker-9] ERROR org.quartz.core.ErrorLogger - Job (async group.refresh_pools_94cec021-4dd2-45ff-9979-1554381bab9e threw an exception. org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1] at org.quartz.core.JobRunShell.run(JobRunShell.java:210) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546) Caused by: org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1 at org.hibernate.jdbc.Expectations$BasicExpectation.checkBatched(Expectations.java:85) at org.hibernate.jdbc.Expectations$BasicExpectation.verifyOutcome(Expectations.java:70) at org.hibernate.jdbc.BatchingBatcher.checkRowCounts(BatchingBatcher.java:90) at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70) at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268) at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:114) at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:109) at org.hibernate.jdbc.AbstractBatcher.prepareBatchStatement(AbstractBatcher.java:244) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:2497) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:2715) at org.hibernate.action.EntityDeleteAction.execute(EntityDeleteAction.java:97) at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:172) at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321) at org.hibernate.event.def.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:64) at org.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:997) at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1590) at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:306) at org.candlepin.model.PoolCurator.entitlementsIn(PoolCurator.java:289) at org.candlepin.model.PoolCurator$$EnhancerByGuice$$1de1355a.CGLIB$entitlementsIn$10(<generated>) at org.candlepin.model.PoolCurator$$EnhancerByGuice$$1de1355a$$FastClassByGuice$$9e2a3eaa.invoke(<generated>) at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.wideplay.warp.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.google.inject.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44) at org.candlepin.model.PoolCurator$$EnhancerByGuice$$1de1355a.entitlementsIn(<generated>) at org.candlepin.controller.CandlepinPoolManager.deletePool(CandlepinPoolManager.java:774) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7.CGLIB$deletePool$9(<generated>) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7$$FastClassByGuice$$633c9893.invoke(<generated>) at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.wideplay.warp.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.google.inject.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7.deletePool(<generated>) at org.candlepin.controller.CandlepinPoolManager.refreshPools(CandlepinPoolManager.java:186) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7.CGLIB$refreshPools$2(<generated>) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7$$FastClassByGuice$$633c9893.invoke(<generated>) at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.ProxyFactory$1.intercept(ProxyFactory.java:53) at org.candlepin.controller.CandlepinPoolManager$$EnhancerByGuice$$4a909b7.refreshPools(<generated>) at org.candlepin.pinsetter.tasks.RefreshPoolsJob.execute(RefreshPoolsJob.java:66) at org.candlepin.pinsetter.tasks.RefreshPoolsJob$$EnhancerByGuice$$fbd1fef5.CGLIB$execute$0(<generated>) at org.candlepin.pinsetter.tasks.RefreshPoolsJob$$EnhancerByGuice$$fbd1fef5$$FastClassByGuice$$4ebc4602.invoke(<generated>) at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.wideplay.warp.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:68) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.google.inject.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44) at org.candlepin.pinsetter.tasks.RefreshPoolsJob$$EnhancerByGuice$$fbd1fef5.execute(<generated>) at org.candlepin.pinsetter.core.TransactionalPinsetterJob.execute(TransactionalPinsetterJob.java:50) at org.quartz.core.JobRunShell.run(JobRunShell.java:199) ... 1 more Devan and I have been trying to reproduce the deadlock, and haven't been able to.
I have only seen it once, but not when I was able to attach a debugger and figure out what was going on.
As far as the other exceptions go, relating to stale objects and batch exceptions, you can at least minimize them by getting the result from
CpClient.put("/owners/#{ORG}/subscriptions", {})
and polling the url it contains until the result is finished.
There's an example of this in candlepin_api.rb in the candlepin git repo
Now, if you do know you're going to call a delete owner right after deleting the owner's subscriptions, you can actually just skip the PUT on subscriptions. owner deletion will handle that for you.
We can wait for it to finish in scope of one process. This will temporary help with one of our scenarios. However, I'm quite afraid it might occur again when dealing with concurrent actions happening in more threads/requests. Waiting for updating in one thread doesn't prevent other thread to work with CP. What if we concurrently run deleting more organizations through some shell script? Script FAILED because manifest is too old. Will look for a more recent one to try. Script FAILED because manifest is too old. Will look for a more recent one to try. Far too old to consider. |
Created attachment 560494 [details] script for reproductin race condition Description of problem: When deleting organization on Katello side, we delete all the content assigned to the owner (including) in CP (products, content, pools, subscriptions...). It works good most of the time, but sometimes it freezes on calling DELETE owner. CP version: candlepin-0.5.16-1 The scenario is: 1. create owner 2. import manifest 3. delete everything It seems, that this is caused by deleting a pool after the owner itself: org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [org.candlepin.model.Pool#4028fa8135617e4601356180103d0426] at org.hibernate.persister.entity.AbstractEntityPersister.check(AbstractEntityPersister.java:1782) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:2541) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:2715) at org.hibernate.action.EntityDeleteAction.execute(EntityDeleteAction.java:97) at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:172) at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321) at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:50) at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1028) at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:366) at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:137) at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:54) at com.wideplay.warp.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:88) at com.google.inject.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64) at com.google.inject.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44) at org.candlepin.pinsetter.tasks.RefreshPoolsJob$$EnhancerByGuice$$daa7d6b4.execute(<generated>) at org.candlepin.pinsetter.core.TransactionalPinsetterJob.execute(TransactionalPinsetterJob.java:50) at org.quartz.core.JobRunShell.run(JobRunShell.java:199) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546) Feb 09 10:44:38 [QuartzScheduler_Worker-12] DEBUG org.candlepin.controller.CandlepinPoolManager - Refreshing pools Feb 09 10:44:38 [QuartzScheduler_Worker-12] DEBUG org.candlepin.controller.CandlepinPoolManager - Found subscriptions: Feb 09 10:44:38 [QuartzScheduler_Worker-12] DEBUG org.candlepin.controller.CandlepinPoolManager - Found pools: Feb 09 10:44:38 [QuartzScheduler_Worker-12] DEBUG org.candlepin.controller.CandlepinPoolManager - EntitlementPool [id = 4028fa8135617e460135618010350422, owner = 4028fa8135617e46013561800e690415, products = ZOO - [org.candlepin.model.ProvidedProduct@542de48f], sub = 4028fa8135617e46013561801025041f, quantity = 20, expires = 2025-05-29 02:00:00.0] Feb 09 10:44:38 [QuartzScheduler_Worker-12] DEBUG org.candlepin.controller.CandlepinPoolManager - EntitlementPool [id = 4028fa8135617e4601356180103d0426, owner = 4028fa8135617e46013561800e690415, products = NATURE - [], sub = 4028fa8135617e46013561800ff8041c, quantity = 30, expires = 2025-05-29 02:00:00.0] Feb 09 10:44:38 [http-8443-2] INFO org.candlepin.resource.OwnerResource - Cleaning up owner: Owner [name = test, key = test, id = 4028fa8135617e46013561800e690415] Feb 09 10:44:38 [http-8443-2] INFO org.candlepin.resource.OwnerResource - Deleting pool: EntitlementPool [id = 4028fa8135617e460135618010350422, owner = 4028fa8135617e46013561800e690415, products = ZOO - [org.candlepin.model.ProvidedProduct@c934a63], sub = 4028fa8135617e46013561801025041f, quantity = 20, expires = 2025-05-29 02:00:00.0] It doesn't occur when using http basic auth, only with OAuth (I assume that it's faster and therefore it causes the workers deleting pools concurrently). I attach a script reproducing this problem. It doesn't occur immediately, but after enough loops it should reproduce this problem. See README