Bug 1366798
| Summary: | Manifest fails to import. | |||
|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Dylan Gross <dgross> | |
| Component: | Subscription Management | Assignee: | Justin Sherrill <jsherril> | |
| Status: | CLOSED ERRATA | QA Contact: | jcallaha | |
| Severity: | high | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 6.2.0 | CC: | abalakht, bbuckingham, bcourt, bkearney, cduryee, jcallaha, jsherril, mmccune, mstead, pmorey, tasander, xdmoon, zhunting | |
| Target Milestone: | Unspecified | Keywords: | Reopened, Triaged | |
| Target Release: | Unused | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | candlepin-0.9.54.9-1 tfm-rubygem-katello-3.0.0.115-1 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1369184 1370205 1426388 (view as bug list) | Environment: | ||
| Last Closed: | 2017-05-01 14:28:02 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1369184 | |||
| Bug Blocks: | 1385841, 1426388 | |||
The error was due to bad debug certificate data that was caused by an old debug cert download bug. Right now, I'm not sure how the customer got into this state. The data doesn't appear to be in a state that I've seen in other reported bug. More investigation will be required to nail down what happened. WORK AROUND =========== Running the following SQL will clean up the bad data and import should start functioning again. NOTE: Debug certificates, if in use, should be manually regenerated per org. SQL TO CLEAN OUT ALL CANDLEPIN DEBUG CERT DATA FOR ALL ORGs =========================================================== delete from cp_subscription where id in (select s.id from cp_subscription s inner join cp_product p on s.product_id = p.id left outer join cp_pool pool on pool.productid = p.id where p.name LIKE '%_ueber_product'); delete from cp_ent_certificate where entitlement_id in (select e.id from cp_entitlement e inner join cp_pool p on e.pool_id = p.id inner join cp_product prod on prod.id = p.productid where prod.name like '%ueber%'); delete from cp_entitlement where pool_id in (select p.id from cp_pool p inner join cp_product prod on prod.id = p.productid where prod.name like '%ueber%'); delete from cp_pool where id in (select p.id from cp_pool p inner join cp_product prod on prod.id = p.productid where prod.name like '%ueber%'); delete from cp_content where id in (select c.id from cp_content c inner join cp_product_content pc on pc.content_id=c.id inner join cp_product p on p.id=pc.product_id left outer join cp_pool pool on pool.productid = p.id where p.name LIKE '%_ueber_product'); delete from cp_product where name like '%_ueber_product'; delete from cp_consumer where name like '%ueber%'; I confirmed that if the ueber cert is removed or updated, katello should figure things out and automatically fix affected capsule repositories upon next cpasule sync without user input. The instructions in #5 should do the trick for workaround. Verified in Satellite 6.2.2 Async Unable to reproduce this issue with the latest version of satellite. Started with two systems, one using default, the other using custom certs. Upgraded the systems and didn't encounter the issue after 10 iterations of downloading the ueber certificates and deleting/uploading/refreshing the manifests. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1993 Upstream bug assigned to jsherril Upstream bug assigned to jsherril Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/16978 has been resolved. Please add verifications steps for this bug to help QE verify Verified in Satellite 6.2.9 Snap 2 Created a new organization called 'testerr' and monitored the candlepin logs from that period. Here are the ueber specific logs 2017-04-06 13:47:15,367 [thread=http-8443-3] [req=b118cd76-6b65-42e3-a8bf-65bcd66bad8a, org=] INFO org.candlepin.common.filter.LoggingFilter - Request: verb=GET, uri=/candlepin/owners/testerr/uebercert 2017-04-06 13:47:15,456 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=] INFO org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/owners/testerr/uebercert 2017-04-06 13:47:15,523 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO org.candlepin.policy.js.pool.PoolRules - Creating new master pool: Pool<type=NORMAL, product=14915008354908818526412, productName=testerr_ueber_product, id=null, quantity=1> full logs from that period 2017-04-06 13:47:14,208 [thread=http-8443-3] [req=c01a99ad-5e81-447f-805d-522077470494, org=] INFO org.candlepin.common.filter.LoggingFilter - Request: verb=GET, uri=/candlepin/status 2017-04-06 13:47:14,212 [thread=http-8443-3] [req=c01a99ad-5e81-447f-805d-522077470494, org=] INFO org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=4 2017-04-06 13:47:14,262 [thread=http-8443-3] [req=4105d2af-42cd-46a4-b582-dbb66fdbe268, org=] INFO org.candlepin.common.filter.LoggingFilter - Request: verb=GET, uri=/candlepin/status 2017-04-06 13:47:14,266 [thread=http-8443-3] [req=4105d2af-42cd-46a4-b582-dbb66fdbe268, org=] INFO org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=4 2017-04-06 13:47:15,101 [thread=http-8443-3] [req=35b594e3-1ada-4c06-91a9-1e5d0d6c694d, org=] INFO org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/owners/ 2017-04-06 13:47:15,133 [thread=http-8443-3] [req=35b594e3-1ada-4c06-91a9-1e5d0d6c694d, org=] INFO org.candlepin.resource.OwnerResource - Created owner: Owner [id: 8a90848f5b3f84cd015b4460a133049f, key: testerr] 2017-04-06 13:47:15,134 [thread=http-8443-3] [req=35b594e3-1ada-4c06-91a9-1e5d0d6c694d, org=] INFO org.candlepin.audit.HornetqEventDispatcher - Sending event: Event [id=null, target=OWNER, type=CREATED, time=Thu Apr 06 13:47:15 EDT 2017, entity=8a90848f5b3f84cd015b4460a133049f] 2017-04-06 13:47:15,140 [thread=http-8443-3] [req=35b594e3-1ada-4c06-91a9-1e5d0d6c694d, org=] INFO org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=39 2017-04-06 13:47:15,235 [thread=http-8443-3] [req=2dbfc60d-4a07-441b-8f81-ed2cfd00123f, org=] INFO org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/owners/testerr/environments 2017-04-06 13:47:15,284 [thread=http-8443-3] [req=2dbfc60d-4a07-441b-8f81-ed2cfd00123f, org=testerr] INFO org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=49 2017-04-06 13:47:15,367 [thread=http-8443-3] [req=b118cd76-6b65-42e3-a8bf-65bcd66bad8a, org=] INFO org.candlepin.common.filter.LoggingFilter - Request: verb=GET, uri=/candlepin/owners/testerr/uebercert 2017-04-06 13:47:15,406 [thread=http-8443-3] [req=b118cd76-6b65-42e3-a8bf-65bcd66bad8a, org=testerr] INFO org.candlepin.common.filter.LoggingFilter - Response: status=404, content-type="application/json", time=39 2017-04-06 13:47:15,456 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=] INFO org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/owners/testerr/uebercert 2017-04-06 13:47:15,521 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO org.candlepin.policy.js.pool.PoolRules - Checking if pools need to be created for: Subscription [id = 8a90848f5b3f84cd015b4460a2b004a1, product = 14915008354908818526412, owner = testerr] 2017-04-06 13:47:15,523 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO org.candlepin.policy.js.pool.PoolRules - Creating new master pool: Pool<type=NORMAL, product=14915008354908818526412, productName=testerr_ueber_product, id=null, quantity=1> 2017-04-06 13:47:15,592 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO org.candlepin.controller.CandlepinPoolManager - Locking pool 8a90848f5b3f84cd015b4460a2c304a2 2017-04-06 13:47:15,596 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO org.candlepin.controller.CandlepinPoolManager - Running pre-entitlement rules. 2017-04-06 13:47:15,605 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO org.candlepin.controller.CandlepinPoolManager - Processing entitlement. 2017-04-06 13:47:15,605 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO org.candlepin.controller.CandlepinPoolManager - Persisting entitlement. 2017-04-06 13:47:15,650 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO org.candlepin.controller.CandlepinPoolManager - Getting subscription: 8a90848f5b3f84cd015b4460a2b004a1 2017-04-06 13:47:15,662 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO org.candlepin.controller.CandlepinPoolManager - Got subscription 2017-04-06 13:47:15,662 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Generating entitlement cert. 2017-04-06 13:47:16,997 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Creating X509 cert. 2017-04-06 13:47:17,021 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Getting PEM encoded cert. 2017-04-06 13:47:17,023 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Persisting cert. 2017-04-06 13:47:17,092 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO org.candlepin.controller.CandlepinPoolManager - Granted entitlement: 8a90848f5b3f84cd015b4460a31604a5 from pool: 8a90848f5b3f84cd015b4460a2c304a2 2017-04-06 13:47:17,101 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO org.candlepin.audit.HornetqEventDispatcher - Sending event: Event [id=null, target=POOL, type=CREATED, time=Thu Apr 06 13:47:15 EDT 2017, entity=8a90848f5b3f84cd015b4460a2c304a2] 2017-04-06 13:47:17,110 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO org.candlepin.audit.HornetqEventDispatcher - Sending event: Event [id=null, target=COMPLIANCE, type=CREATED, time=Thu Apr 06 13:47:15 EDT 2017, entity=8a90848f5b3f84cd015b4460a2e504a4] 2017-04-06 13:47:17,116 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=1660 2017-04-06 13:47:19,071 [thread=http-8443-3] [req=403d2066-e3b2-4e60-ae5a-9886d75e6bab, org=] INFO org.candlepin.common.filter.LoggingFilter - Request: verb=GET, uri=/candlepin/pools/8a90848f5b3f84cd015b4460a2c304a2 2017-04-06 13:47:19,114 [thread=http-8443-3] [req=403d2066-e3b2-4e60-ae5a-9886d75e6bab, org=testerr] INFO org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=43 2017-04-06 13:47:19,173 [thread=http-8443-3] [req=5cb729e1-a26c-4881-b200-c7ac5399db60, org=] INFO org.candlepin.common.filter.LoggingFilter - Request: verb=GET, uri=/candlepin/activation_keys/?include=id&include=pools.pool.id 2017-04-06 13:47:19,188 [thread=http-8443-3] [req=5cb729e1-a26c-4881-b200-c7ac5399db60, org=] INFO org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=15 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1191 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1191 |
Description of problem: Importing of manifest via webui or hammer fails Version-Release number of selected component (if applicable): Red Hat Satellite 6.2.0 How reproducible: Unknown Steps to Reproduce: Customer encountered when removing manifest (successfully), and then attempting to re-attach the manifest. Actual results: WebUI screen will spin indefinitely. The /var/log/candlepin/candlepin.log generates a long trace beginning with ... 2016-08-08 15:10:01,128 [thread=http-bio-8443-exec-4] [req=9156e50d-852d-4256-b49e-8493bd9f1978, org=OrgName] INFO org.candlepin.resource.OwnerResource - Import attempt completed for owner OrgName 2016-08-08 15:10:01,135 [thread=http-bio-8443-exec-4] [req=9156e50d-852d-4256-b49e-8493bd9f1978, org=OrgName] ERROR org.candlepin.common.exceptions.mappers.CandlepinExceptionMapper - Runtime Error query did not return a unique result: 2 at org.hibernate.internal.AbstractQueryImpl.uniqueElement:914 org.hibernate.NonUniqueResultException: query did not return a unique result: 2 at org.hibernate.internal.AbstractQueryImpl.uniqueElement(AbstractQueryImpl.java:914) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final] at org.hibernate.internal.CriteriaImpl.uniqueResult(CriteriaImpl.java:396) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final] at org.candlepin.model.PoolCurator.findUeberPool(PoolCurator.java:334) ~[PoolCurator.class:na] at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) ~[guice-persist-3.0.jar:na] at org.candlepin.controller.CandlepinPoolManager.findUeberPool(CandlepinPoolManager.java:1640) ~[CandlepinPoolManager.class:na] at org.candlepin.controller.CandlepinPoolManager.refreshPoolsWithRegeneration(CandlepinPoolManager.java:173) ~[CandlepinPoolManager.class:na] at org.candlepin.controller.Refresher.run(Refresher.java:110) ~[Refresher.class:na] at org.candlepin.sync.Importer.importObjects(Importer.java:450) ~[Importer.class:na] at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:66) ~[guice-persist-3.0.jar:na] at org.candlepin.sync.Importer.loadExport(Importer.java:291) ~[Importer.class:na] Expected results: The manifest imports successfully. Additional info: Note, while the actual error in the log complains about a non-unique result similar to BZ#1337833, the trace actually begins to differ around the 3rd line when it calls "org.candlepin.model.PoolCurator.findUeberPool(PoolCurator.java:334) ~[PoolCurator.class:na]: