Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
(this was seen once at customer on 6.3, harder to reproduce, but it is expected to be present in any Sat/candlepin version)
Manifest refresh / import fails with error:
javax.persistence.PersistenceException: org.hibernate.PersistentObjectException: detached entity passed to persist: org.candlepin.model.EntitlementCertificate
candlepin's error log contains that error both for the import request and for ExpiredPoolsJob job. The cause is, the job was removing pools at the same time when import task was doing so:
2019-05-06 08:59:50,474 [thread=http-bio-8443-exec-69] [req=34af4f3a-5fab-42b5-b957-fbe0f8ad0b7e, org=, csid=92b17d79] INFO org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/owners/ORG/imports
..
2019-05-06 09:03:42,688 [thread=http-bio-8443-exec-69] [req=34af4f3a-5fab-42b5-b957-fbe0f8ad0b7e, org=ORG, csid=92b17d79] INFO org.candlepin.controller.CandlepinPoolManager - Recomputing status for 475 consumers
2019-05-06 09:03:52,319 [thread=http-bio-8443-exec-69] [req=34af4f3a-5fab-42b5-b957-fbe0f8ad0b7e, org=ORG, csid=92b17d79] ERROR org.candlepin.sync.Importer - Failed to import archive
javax.persistence.PersistenceException: org.hibernate.PersistentObjectException: detached entity passed to persist: org.candlepin.model.EntitlementCertificate
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1608) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.flush(AbstractEntityManagerImpl.java:1303) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]
at org.candlepin.model.AbstractHibernateCurator.flush(AbstractHibernateCurator.java:564) ~[AbstractHibernateCurator.class:na]
at org.candlepin.controller.CandlepinPoolManager.deletePools(CandlepinPoolManager.java:2245) ~[CandlepinPoolManager.class:na]
at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) ~[guice-persist-3.0.jar:na]
at org.candlepin.controller.CandlepinPoolManager.deletePools(CandlepinPoolManager.java:1998) ~[CandlepinPoolManager.class:na]
at org.candlepin.controller.CandlepinPoolManager.refreshPoolsWithRegeneration(CandlepinPoolManager.java:359) ~[CandlepinPoolManager.class:na]
at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) ~[guice-persist-3.0.jar:na]
at org.candlepin.controller.Refresher.run(Refresher.java:159) ~[Refresher.class:na]
at org.candlepin.sync.Importer.importObjects(Importer.java:595) [Importer.class:na]
at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:66) ~[guice-persist-3.0.jar:na]
at org.candlepin.sync.Importer.doExport(Importer.java:415) [Importer.class:na]
at org.candlepin.sync.Importer.loadExport(Importer.java:176) [Importer.class:na]
..
while also:
2019-05-06 09:03:57,076 [thread=QuartzScheduler_Worker-15] [job=ExpiredPoolsJob-76d0e223-d4ee-421e-ad17-f07925073177, org=, csid=] INFO org.candlepin.controller.CandlepinPoolManager - Recomputing status for 475 consumers
2019-05-06 09:04:05,497 [thread=QuartzScheduler_Worker-15] [job=ExpiredPoolsJob-76d0e223-d4ee-421e-ad17-f07925073177, org=, csid=] ERROR org.candlepin.pinsetter.tasks.KingpinJob - Persistence exception caught running pinsetter task. Attempt: 0, Refire: true
javax.persistence.PersistenceException: org.hibernate.PersistentObjectException: detached entity passed to persist: org.candlepin.model.EntitlementCertificate
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1608) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.flush(AbstractEntityManagerImpl.java:1303) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]
at org.candlepin.model.AbstractHibernateCurator.flush(AbstractHibernateCurator.java:564) ~[AbstractHibernateCurator.class:na]
at org.candlepin.controller.CandlepinPoolManager.deletePools(CandlepinPoolManager.java:2245) ~[CandlepinPoolManager.class:na]
at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) ~[guice-persist-3.0.jar:na]
at org.candlepin.controller.CandlepinPoolManager.deletePools(CandlepinPoolManager.java:1998) ~[CandlepinPoolManager.class:na]
at org.candlepin.controller.CandlepinPoolManager.cleanupExpiredPoolsImpl(CandlepinPoolManager.java:508) ~[CandlepinPoolManager.class:na]
at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:66) ~[guice-persist-3.0.jar:na]
at org.candlepin.controller.CandlepinPoolManager.cleanupExpiredPools(CandlepinPoolManager.java:480) ~[CandlepinPoolManager.class:na]
at org.candlepin.pinsetter.tasks.ExpiredPoolsJob.toExecute(ExpiredPoolsJob.java:49) ~[ExpiredPoolsJob.class:na]
Version-Release number of selected component (if applicable):
- seen in Sat 6.3.5 / candlepin-2.1.24-1.el7.noarch
- expected to be in any Sat/candlepin version
How reproducible:
low probability
Steps to Reproduce:
1. manually increase frequency of the job execution every 10s:
- add to /etc/candlepin/candlepin.conf :
pinsetter.org.candlepin.pinsetter.tasks.ExpiredPoolsJob.schedule=10/1 * * * * ?
- restart tomcat service
2. Have some manifest in Satellite with some certs that has expired
3. Refresh the manifest
Actual results:
3. with some probability, the refresh fails with above error in candlepin logs
Expected results:
3. manifest refresh succeeds
Additional info:
In case some more specific or reliable reproducer is needed, let me know and I can try to find it out. Currently I think that:
- the cause of the bug is evident (there should be a lock in calling org.candlepin.controller.CandlepinPoolManager.deletePools)
- devel fixing this would know particular better reproducer better than me, I guess
But I can play with reproducer, if needed..
Hi, the expired pools job & the refresh job can not run concurrently as they are both modifying the same data. In 6.3 the expired pools job could take a long time to run and it has been improved in later versions of Candlepin in order to reduce the amount of time that the overlap occurs.
(In reply to Barnaby Court from comment #4)
> Hi, the expired pools job & the refresh job can not run concurrently as they
> are both modifying the same data.
But an user does not know when/if expired pools job is running, and it should be candlepin's duty to prevent the concurrency issue.
> In 6.3 the expired pools job could take a
> long time to run and it has been improved in later versions of Candlepin in
> order to reduce the amount of time that the overlap occurs.
So the probability to hit the bug has been decreased "only". If you see that as sufficient, ok. In the worst case, we will reopen the bug.
Description of problem: (this was seen once at customer on 6.3, harder to reproduce, but it is expected to be present in any Sat/candlepin version) Manifest refresh / import fails with error: javax.persistence.PersistenceException: org.hibernate.PersistentObjectException: detached entity passed to persist: org.candlepin.model.EntitlementCertificate candlepin's error log contains that error both for the import request and for ExpiredPoolsJob job. The cause is, the job was removing pools at the same time when import task was doing so: 2019-05-06 08:59:50,474 [thread=http-bio-8443-exec-69] [req=34af4f3a-5fab-42b5-b957-fbe0f8ad0b7e, org=, csid=92b17d79] INFO org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/owners/ORG/imports .. 2019-05-06 09:03:42,688 [thread=http-bio-8443-exec-69] [req=34af4f3a-5fab-42b5-b957-fbe0f8ad0b7e, org=ORG, csid=92b17d79] INFO org.candlepin.controller.CandlepinPoolManager - Recomputing status for 475 consumers 2019-05-06 09:03:52,319 [thread=http-bio-8443-exec-69] [req=34af4f3a-5fab-42b5-b957-fbe0f8ad0b7e, org=ORG, csid=92b17d79] ERROR org.candlepin.sync.Importer - Failed to import archive javax.persistence.PersistenceException: org.hibernate.PersistentObjectException: detached entity passed to persist: org.candlepin.model.EntitlementCertificate at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1608) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.jpa.spi.AbstractEntityManagerImpl.flush(AbstractEntityManagerImpl.java:1303) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final] at org.candlepin.model.AbstractHibernateCurator.flush(AbstractHibernateCurator.java:564) ~[AbstractHibernateCurator.class:na] at org.candlepin.controller.CandlepinPoolManager.deletePools(CandlepinPoolManager.java:2245) ~[CandlepinPoolManager.class:na] at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) ~[guice-persist-3.0.jar:na] at org.candlepin.controller.CandlepinPoolManager.deletePools(CandlepinPoolManager.java:1998) ~[CandlepinPoolManager.class:na] at org.candlepin.controller.CandlepinPoolManager.refreshPoolsWithRegeneration(CandlepinPoolManager.java:359) ~[CandlepinPoolManager.class:na] at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) ~[guice-persist-3.0.jar:na] at org.candlepin.controller.Refresher.run(Refresher.java:159) ~[Refresher.class:na] at org.candlepin.sync.Importer.importObjects(Importer.java:595) [Importer.class:na] at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:66) ~[guice-persist-3.0.jar:na] at org.candlepin.sync.Importer.doExport(Importer.java:415) [Importer.class:na] at org.candlepin.sync.Importer.loadExport(Importer.java:176) [Importer.class:na] .. while also: 2019-05-06 09:03:57,076 [thread=QuartzScheduler_Worker-15] [job=ExpiredPoolsJob-76d0e223-d4ee-421e-ad17-f07925073177, org=, csid=] INFO org.candlepin.controller.CandlepinPoolManager - Recomputing status for 475 consumers 2019-05-06 09:04:05,497 [thread=QuartzScheduler_Worker-15] [job=ExpiredPoolsJob-76d0e223-d4ee-421e-ad17-f07925073177, org=, csid=] ERROR org.candlepin.pinsetter.tasks.KingpinJob - Persistence exception caught running pinsetter task. Attempt: 0, Refire: true javax.persistence.PersistenceException: org.hibernate.PersistentObjectException: detached entity passed to persist: org.candlepin.model.EntitlementCertificate at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1608) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.jpa.spi.AbstractEntityManagerImpl.flush(AbstractEntityManagerImpl.java:1303) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final] at org.candlepin.model.AbstractHibernateCurator.flush(AbstractHibernateCurator.java:564) ~[AbstractHibernateCurator.class:na] at org.candlepin.controller.CandlepinPoolManager.deletePools(CandlepinPoolManager.java:2245) ~[CandlepinPoolManager.class:na] at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) ~[guice-persist-3.0.jar:na] at org.candlepin.controller.CandlepinPoolManager.deletePools(CandlepinPoolManager.java:1998) ~[CandlepinPoolManager.class:na] at org.candlepin.controller.CandlepinPoolManager.cleanupExpiredPoolsImpl(CandlepinPoolManager.java:508) ~[CandlepinPoolManager.class:na] at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:66) ~[guice-persist-3.0.jar:na] at org.candlepin.controller.CandlepinPoolManager.cleanupExpiredPools(CandlepinPoolManager.java:480) ~[CandlepinPoolManager.class:na] at org.candlepin.pinsetter.tasks.ExpiredPoolsJob.toExecute(ExpiredPoolsJob.java:49) ~[ExpiredPoolsJob.class:na] Version-Release number of selected component (if applicable): - seen in Sat 6.3.5 / candlepin-2.1.24-1.el7.noarch - expected to be in any Sat/candlepin version How reproducible: low probability Steps to Reproduce: 1. manually increase frequency of the job execution every 10s: - add to /etc/candlepin/candlepin.conf : pinsetter.org.candlepin.pinsetter.tasks.ExpiredPoolsJob.schedule=10/1 * * * * ? - restart tomcat service 2. Have some manifest in Satellite with some certs that has expired 3. Refresh the manifest Actual results: 3. with some probability, the refresh fails with above error in candlepin logs Expected results: 3. manifest refresh succeeds Additional info: