Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
If user performs a manifest refresh against a manifest having all expired subscriptions or an empty manifest, Candlepin will remove all the environment contents in the organization. After renewing all subscriptions and attach to the manifest, user performs a manifest refresh again. The job completed successfully but all clients are not able to subscribe to or enable any Redhat repositories due to no content listing.
Steps to Reproduce:
1. Have a valid manifest imported to the Satellite. Enable and sync some Redhat repositories. Create some custom repositories. Create one or more content views and attach some repositories.
2. Go to Web UI -> Content -> Subscriptions -> Manage Manifest -> click the "Subscription Allocation" link to go to the Customer Portal page.
3. In the Customer Portal, delete all subscriptions allocated to the manifest.
4. Then refresh the manifest using the Satellite Web UI.
Actual results:
# You will see the import job remove all products and contents references like below in candlepin.log file.
NOTE: If you don't see the "Removing owner-content references for owner" and "Removing owner-product references for owner", you will need to follow step (4) again. I refreshed the empty manifest twice to reproduce the issue. Not sure why.
--------------------------------------------------------------------------
2022-03-09 14:41:29,987 [thread=http-bio-127.0.0.1-23443-exec-3] [req=67b1d364-ddd4-4c72-a2f6-09291d658b8d, org=, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/owners/redhat/imports/async
2022-03-09 14:41:30,000 [thread=http-bio-127.0.0.1-23443-exec-3] [req=67b1d364-ddd4-4c72-a2f6-09291d658b8d, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.resource.OwnerResource - Running async import of archive /var/cache/tomcat/temp/pfx11102612137104972728sfx for owner redhat
2022-03-09 14:41:30,036 [thread=http-bio-127.0.0.1-23443-exec-3] [req=67b1d364-ddd4-4c72-a2f6-09291d658b8d, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.async.JobManager - Job queued: AsyncJobStatus [id: 8ac8a52d7f673a55017f6cfb68ad01c0, name: Import Manifest, key: ImportJob, state: QUEUED]
2022-03-09 14:41:30,046 [thread=http-bio-127.0.0.1-23443-exec-3] [req=67b1d364-ddd4-4c72-a2f6-09291d658b8d, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=60
2022-03-09 14:41:30,063 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.async.JobManager - Starting job "Import Manifest" using class: org.candlepin.async.tasks.ImportJob
2022-03-09 14:41:30,101 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.sync.RulesImporter - Importing new rules from manifest, current version: 5.41 new version: 5.41
2022-03-09 14:41:30,129 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.policy.js.JsRunnerProvider - Recompiling rules with timestamp: 2022-03-09 14:41:30.103
2022-03-09 14:41:32,583 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] WARN org.candlepin.sync.Importer - No products found to import, skipping product import.
2022-03-09 14:41:32,585 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] WARN org.candlepin.sync.Importer - No entitlements in manifest, removing all subscriptions for owner.
2022-03-09 14:41:32,589 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.controller.CandlepinPoolManager - Refreshing pools for owner: Owner [id: 8ac8a52d7f1ba631017f1bb6f3ed0001, key: redhat]
2022-03-09 14:41:32,851 [thread=http-bio-127.0.0.1-23443-exec-7] [req=634aef21-933c-4cd6-adee-466ae61108c4, org=, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.common.filter.LoggingFilter - Request: verb=GET, uri=/candlepin/jobs/8ac8a52d7f673a55017f6cfb68ad01c0?
2022-03-09 14:41:32,856 [thread=http-bio-127.0.0.1-23443-exec-7] [req=634aef21-933c-4cd6-adee-466ae61108c4, org=, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=5
2022-03-09 14:41:33,641 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.model.OwnerContentCurator - Removing owner-content references for owner: Owner [id: 8ac8a52d7f1ba631017f1bb6f3ed0001, key: redhat], [8ac8a52d7f1ba631017f1bc30ef50dbf, 8ac8a52d7f1ba631017f1bc30ef50dbb, 8ac8a52d7f1ba631017f1bc30ef50dbc, 8ac8a52d7f1ba631017f1bc30ef50dbd, 8ac8a52d7f1ba631017f1bc30ef50dbe, 8ac8a52d7f1ba631017f1bc30ef50dca, 8ac8a52d7f1ba631017f1bc310653139, 8ac8a52d7f1ba631017f1bc310653138, 8ac8a52d7f1ba631017f1bc310653137, 8ac8a52d7f1ba631017f1bc310653136, 8ac8a52d7f673a55017f675abd700080, 8ac8a52d7f1ba631017f1bc310653135, 8ac8a52d7f1ba631017f1bc310653134, 8ac8a52d7f1ba631017f1bc30e77045f, 8ac8a52d7f1ba631017f1bc31065313b, 8ac8a52d7f1ba631017f1bc31065313a, 8ac8a52d7f1ba631017f1bc30ffc28a1, 8ac8a52d7f1ba631017f1bc30ffc28a0, 8ac8a52d7f1ba631017f1bc31065313f, 8ac8a52d7f1ba631017f1bc31065313e, 8ac8a52d7f1ba631017f1bc31065313d, 8ac8a52d7f1ba631017f1bc31065313c, 8ac8a52d7f1ba631017f1bc30e770465, 8ac8a52d7f1ba631017f1bc30e770466, 8ac8a52d7f1ba631017f1bc30ef50db9, 8ac8a52d7f1ba631017f1bc310653140, 8ac8a52d7f1ba631017f1bc30ef50dc0, 8ac8a52d7f1ba631017f1bc30ef50dc1, 8ac8a52d7f1ba631017f1bc3102d2ce7, 8ac8a52d7f1ba631017f1bc30ef50dc2, 8ac8a52d7f1ba631017f1bc3102d2ce5, 8ac8a52d7f1ba631017f1bc3102d2ce6, 8ac8a52d7f1ba631017f1bc3102d2ce3, 8ac8a52d7f1ba631017f1bc3102d2ce4, 8ac8a52d7f1ba631017f1bc30e770461, 8ac8a52d7f1ba631017f1bc3102d2ce1, 8ac8a52d7f1ba631017f1bc30e770462, 8ac8a52d7f1ba631017f1bc3102d2ce2, 8ac8a52d7f1ba631017f1bc30e770463, 8ac8a52d7f1ba631017f1bc3102d2ce0, 8ac8a52d7f1ba631017f1bc30e770464, 8ac8a52d7f1ba631017f1bc30e770460, 8ac8a52d7f673a55017f675abcd0000a, 8ac8a52d7f1ba631017f1bc30ef50dba, 8ac8a52d7f1ba631017f1bc3102d2cca, 8ac8a52d7f1ba631017f1bc30ef50dc7, 8ac8a52d7f1ba631017f1bc30ef50dc8, 8ac8a52d7f1ba631017f1bc30ef50dc9, 8ac8a52d7f1ba631017f1bc30ef50dc3, 8ac8a52d7f1ba631017f1bc30ef50dc4, 8ac8a52d7f1ba631017f1bc30ef50dc5, 8ac8a52d7f1ba631017f1bc30ef50dc6, 8ac8a52d7f1ba631017f1bc30eb80829, 8ac8a52d7f1ba631017f1bc30eb80827, 8ac8a52d7f1ba631017f1bc30eb80828, 8ac8a52d7f1ba631017f1bc30eb8081a,
<snip>
2022-03-09 14:41:35,076 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.model.OwnerProductCurator - Removing owner-product references for owner: Owner [id: 8ac8a52d7f1ba631017f1bb6f3ed0001, key: redhat], [8ac8a52d7f1ba631017f1bc310402ef8, 8ac8a52d7f1ba631017f1bc3108f3782, 8ac8a52d7f1ba631017f1bc30ede0bb3, 8ac8a52d7f1ba631017f1bc30e730431, 8ac8a52d7f1ba631017f1bc3107e34eb, 8ac8a52d7f1ba631017f1bc310a039bd, 8ac8a52d7f1ba631017f1bc30ef80e14, 8ac8a52d7f1ba631017f1bc30eef0d29, 8ac8a52d7f1ba631017f1bc30ee40c42, 8ac8a52d7f1ba631017f1bc30f3a143f, 8ac8a52d7f1ba631017f1bc30ecd0a1a, 8ac8a52d7f1ba631017f1bc3102a2c79, 8ac8a52d7f1ba631017f1bc3108e3742, 8ac8a52d7f1ba631017f1bc310a83af3, 8ac8a52d7f1ba631017f1bc30ed00a67, 8ac8a52d7f1ba631017f1bc30f871be6, 8ac8a52d7f1ba631017f1bc30ff2274c, 8ac8a52d7f1ba631017f1bc3109037ad, 8ac8a52d7f1ba631017f1bc310a83ae5, 8ac8a52d7f1ba631017f1bc310172ab9, 8ac8a52d7f1ba631017f1bc30eb20783, 8ac8a52d7f1ba631017f1bc8da383c8c, 8ac8a52d7f1ba631017f1bc3109037b1, 8ac8a52d7f1ba631017f1bc3100629e9, 8ac8a52d7f1ba631017f1bc30f8f1caf, 8ac8a52d7f1ba631017f1bc310a73ac5, 8ac8a52d7f1ba631017f1bc3102a2c8c, 8ac8a52d7f1ba631017f1bc3108b36bf, 8ac8a52d7f1ba631017f1bc3100b2a44, 8ac8a52d7f1ba631017f1bc30f8f1cc9, 8ac8a52d7f1ba631017f1bc310ac3b65, 8ac8a52d7f1ba631017f1bc30f10100b, 8ac8a52d7f1ba631017f1bc30ef20d82, 8ac8a52d7f1ba631017f1bc310ad3b8e, 8ac8a52d7f1ba631017f1bc30f8f1cc8, 8ac8a52d7f1ba631017f1bc30f4c1577, 8ac8a52d7f1ba631017f1bc30e9b0650, 8ac8a52d7f1ba631017f1bc30ec50964, 8ac8a52d7f1ba631017f1bc30f8d1c94, 8ac8a52d7f1ba631017f1bc30eb507f6, 8ac8a52d7f1ba631017f1bc30f281251, 8ac8a52d7f1ba631017f1bc310a73ad5, 8ac8a52d7f1ba631017f1bc8da373c60,
<snip>
2022-03-09 14:41:35,128 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.model.OwnerProductCurator - 293 owner-product relations removed
2022-03-09 14:41:35,175 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.model.OwnerProductCurator - 0 activation key product(s) removed
2022-03-09 14:41:35,454 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.controller.CandlepinPoolManager - Refresh pools for owner: redhat completed in: 2865ms
2022-03-09 14:41:35,615 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.controller.ManifestManager - Deleting stored manifest file: 8ac8a52d7f673a55017f6cfb689101bf
2022-03-09 14:41:35,714 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.async.tasks.ImportJob - Async import complete.
2022-03-09 14:41:35,794 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.async.JobManager - Job "Import Manifest" completed in 5673ms
--------------------------------------------------------------------------
# Before refreshing the manifest:
su - postgres -c "psql candlepin -c \"select e.name, c.contenturl, c.vendor, c.label from cp2_environment_content ec left join cp_environment e on e.id = ec.environment_id left join cp2_content c on c.uuid = content_uuid order by e.name, c.vendor, c.label;\""
name | contenturl | vendor | label
---------------------------------+-----------------------------------------------------------------+---------+-----------------------------------------
Library | /custom/hao_product/no-feed-repo | Custom | redhat_hao_product_no-feed-repo
Library | /content/dist/rhel/server/7/$releasever/$basearch/os | Red Hat | rhel-7-server-rpms
Library | /content/dist/rhel/server/7/7Server/$basearch/sat-tools/6.10/os | Red Hat | rhel-7-server-satellite-tools-6.10-rpms
Library/cv-local-repos | /custom/hao_product/no-feed-repo | Custom | redhat_hao_product_no-feed-repo
Library/sat-tools-only-imported | /content/dist/rhel/server/7/7Server/$basearch/sat-tools/6.10/os | Red Hat | rhel-7-server-satellite-tools-6.10-rpms
# After refreshing the manifest. Only the custom environment contents are left:
candlepin=# select e.name, c.contenturl, c.vendor, c.label from cp2_environment_content ec left join cp_environment e on e.id = ec.environment_id left join cp2_content c on c.uuid = content_uuid;
name | contenturl | vendor | label
------------------------+----------------------------------+--------+---------------------------------
Library | /custom/hao_product/no-feed-repo | Custom | redhat_hao_product_no-feed-repo
Library/cv-local-repos | /custom/hao_product/no-feed-repo | Custom | redhat_hao_product_no-feed-repo
Adding the subscriptions back and refresh the manifest doesn't solve the issue.
Expected results:
Anything that we can improve the avoid this issue.
The workaround of this issue is:
Steps:
1) Complete resync all Redhat repositories.
2) Publish new version and promote all content views.
3) Publish new version and promote all composite content views if any.
Steps to Retest:
1. Have a valid manifest imported to the Satellite. Enable and sync some Redhat repositories. Create some custom repositories. Create one or more content views and attach some repositories.
2. Go to Web UI -> Content -> Subscriptions -> Manage Manifest -> click the "Subscription Allocation" link to go to the Customer Portal page.
3. In the Customer Portal, delete all subscriptions allocated to the manifest.
4. Then refresh the manifest using the Satellite Web UI.
5. Run grep for "Removing owner-content references for owner" on /var/log/candlepin/candlepin.log
6. Run su - postgres -c "psql candlepin -c \"select e.name, c.contenturl, c.vendor, c.label from cp2_environment_content ec left join cp_environment e on e.id = ec.environment_id left join cp2_content c on c.uuid = content_uuid order by e.name, c.vendor, c.label;\""
Expected result:
5. Should return empty
6. Should return only the number of repos that was synced and added to cv
Actual:
5. Did return empty
6. Did return only the number of repos that was synced and added to cv
I also did manifest refresh a few times to ensure the error didn't occur anymore. Everything seems to check out ok.
Verified on 6.12 snap 4
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Important: Satellite 6.12 Release), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2022:8506
Description of problem: If user performs a manifest refresh against a manifest having all expired subscriptions or an empty manifest, Candlepin will remove all the environment contents in the organization. After renewing all subscriptions and attach to the manifest, user performs a manifest refresh again. The job completed successfully but all clients are not able to subscribe to or enable any Redhat repositories due to no content listing. Steps to Reproduce: 1. Have a valid manifest imported to the Satellite. Enable and sync some Redhat repositories. Create some custom repositories. Create one or more content views and attach some repositories. 2. Go to Web UI -> Content -> Subscriptions -> Manage Manifest -> click the "Subscription Allocation" link to go to the Customer Portal page. 3. In the Customer Portal, delete all subscriptions allocated to the manifest. 4. Then refresh the manifest using the Satellite Web UI. Actual results: # You will see the import job remove all products and contents references like below in candlepin.log file. NOTE: If you don't see the "Removing owner-content references for owner" and "Removing owner-product references for owner", you will need to follow step (4) again. I refreshed the empty manifest twice to reproduce the issue. Not sure why. -------------------------------------------------------------------------- 2022-03-09 14:41:29,987 [thread=http-bio-127.0.0.1-23443-exec-3] [req=67b1d364-ddd4-4c72-a2f6-09291d658b8d, org=, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/owners/redhat/imports/async 2022-03-09 14:41:30,000 [thread=http-bio-127.0.0.1-23443-exec-3] [req=67b1d364-ddd4-4c72-a2f6-09291d658b8d, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.resource.OwnerResource - Running async import of archive /var/cache/tomcat/temp/pfx11102612137104972728sfx for owner redhat 2022-03-09 14:41:30,036 [thread=http-bio-127.0.0.1-23443-exec-3] [req=67b1d364-ddd4-4c72-a2f6-09291d658b8d, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.async.JobManager - Job queued: AsyncJobStatus [id: 8ac8a52d7f673a55017f6cfb68ad01c0, name: Import Manifest, key: ImportJob, state: QUEUED] 2022-03-09 14:41:30,046 [thread=http-bio-127.0.0.1-23443-exec-3] [req=67b1d364-ddd4-4c72-a2f6-09291d658b8d, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=60 2022-03-09 14:41:30,063 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.async.JobManager - Starting job "Import Manifest" using class: org.candlepin.async.tasks.ImportJob 2022-03-09 14:41:30,101 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.sync.RulesImporter - Importing new rules from manifest, current version: 5.41 new version: 5.41 2022-03-09 14:41:30,129 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.policy.js.JsRunnerProvider - Recompiling rules with timestamp: 2022-03-09 14:41:30.103 2022-03-09 14:41:32,583 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] WARN org.candlepin.sync.Importer - No products found to import, skipping product import. 2022-03-09 14:41:32,585 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] WARN org.candlepin.sync.Importer - No entitlements in manifest, removing all subscriptions for owner. 2022-03-09 14:41:32,589 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.controller.CandlepinPoolManager - Refreshing pools for owner: Owner [id: 8ac8a52d7f1ba631017f1bb6f3ed0001, key: redhat] 2022-03-09 14:41:32,851 [thread=http-bio-127.0.0.1-23443-exec-7] [req=634aef21-933c-4cd6-adee-466ae61108c4, org=, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.common.filter.LoggingFilter - Request: verb=GET, uri=/candlepin/jobs/8ac8a52d7f673a55017f6cfb68ad01c0? 2022-03-09 14:41:32,856 [thread=http-bio-127.0.0.1-23443-exec-7] [req=634aef21-933c-4cd6-adee-466ae61108c4, org=, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=5 2022-03-09 14:41:33,641 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.model.OwnerContentCurator - Removing owner-content references for owner: Owner [id: 8ac8a52d7f1ba631017f1bb6f3ed0001, key: redhat], [8ac8a52d7f1ba631017f1bc30ef50dbf, 8ac8a52d7f1ba631017f1bc30ef50dbb, 8ac8a52d7f1ba631017f1bc30ef50dbc, 8ac8a52d7f1ba631017f1bc30ef50dbd, 8ac8a52d7f1ba631017f1bc30ef50dbe, 8ac8a52d7f1ba631017f1bc30ef50dca, 8ac8a52d7f1ba631017f1bc310653139, 8ac8a52d7f1ba631017f1bc310653138, 8ac8a52d7f1ba631017f1bc310653137, 8ac8a52d7f1ba631017f1bc310653136, 8ac8a52d7f673a55017f675abd700080, 8ac8a52d7f1ba631017f1bc310653135, 8ac8a52d7f1ba631017f1bc310653134, 8ac8a52d7f1ba631017f1bc30e77045f, 8ac8a52d7f1ba631017f1bc31065313b, 8ac8a52d7f1ba631017f1bc31065313a, 8ac8a52d7f1ba631017f1bc30ffc28a1, 8ac8a52d7f1ba631017f1bc30ffc28a0, 8ac8a52d7f1ba631017f1bc31065313f, 8ac8a52d7f1ba631017f1bc31065313e, 8ac8a52d7f1ba631017f1bc31065313d, 8ac8a52d7f1ba631017f1bc31065313c, 8ac8a52d7f1ba631017f1bc30e770465, 8ac8a52d7f1ba631017f1bc30e770466, 8ac8a52d7f1ba631017f1bc30ef50db9, 8ac8a52d7f1ba631017f1bc310653140, 8ac8a52d7f1ba631017f1bc30ef50dc0, 8ac8a52d7f1ba631017f1bc30ef50dc1, 8ac8a52d7f1ba631017f1bc3102d2ce7, 8ac8a52d7f1ba631017f1bc30ef50dc2, 8ac8a52d7f1ba631017f1bc3102d2ce5, 8ac8a52d7f1ba631017f1bc3102d2ce6, 8ac8a52d7f1ba631017f1bc3102d2ce3, 8ac8a52d7f1ba631017f1bc3102d2ce4, 8ac8a52d7f1ba631017f1bc30e770461, 8ac8a52d7f1ba631017f1bc3102d2ce1, 8ac8a52d7f1ba631017f1bc30e770462, 8ac8a52d7f1ba631017f1bc3102d2ce2, 8ac8a52d7f1ba631017f1bc30e770463, 8ac8a52d7f1ba631017f1bc3102d2ce0, 8ac8a52d7f1ba631017f1bc30e770464, 8ac8a52d7f1ba631017f1bc30e770460, 8ac8a52d7f673a55017f675abcd0000a, 8ac8a52d7f1ba631017f1bc30ef50dba, 8ac8a52d7f1ba631017f1bc3102d2cca, 8ac8a52d7f1ba631017f1bc30ef50dc7, 8ac8a52d7f1ba631017f1bc30ef50dc8, 8ac8a52d7f1ba631017f1bc30ef50dc9, 8ac8a52d7f1ba631017f1bc30ef50dc3, 8ac8a52d7f1ba631017f1bc30ef50dc4, 8ac8a52d7f1ba631017f1bc30ef50dc5, 8ac8a52d7f1ba631017f1bc30ef50dc6, 8ac8a52d7f1ba631017f1bc30eb80829, 8ac8a52d7f1ba631017f1bc30eb80827, 8ac8a52d7f1ba631017f1bc30eb80828, 8ac8a52d7f1ba631017f1bc30eb8081a, <snip> 2022-03-09 14:41:35,076 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.model.OwnerProductCurator - Removing owner-product references for owner: Owner [id: 8ac8a52d7f1ba631017f1bb6f3ed0001, key: redhat], [8ac8a52d7f1ba631017f1bc310402ef8, 8ac8a52d7f1ba631017f1bc3108f3782, 8ac8a52d7f1ba631017f1bc30ede0bb3, 8ac8a52d7f1ba631017f1bc30e730431, 8ac8a52d7f1ba631017f1bc3107e34eb, 8ac8a52d7f1ba631017f1bc310a039bd, 8ac8a52d7f1ba631017f1bc30ef80e14, 8ac8a52d7f1ba631017f1bc30eef0d29, 8ac8a52d7f1ba631017f1bc30ee40c42, 8ac8a52d7f1ba631017f1bc30f3a143f, 8ac8a52d7f1ba631017f1bc30ecd0a1a, 8ac8a52d7f1ba631017f1bc3102a2c79, 8ac8a52d7f1ba631017f1bc3108e3742, 8ac8a52d7f1ba631017f1bc310a83af3, 8ac8a52d7f1ba631017f1bc30ed00a67, 8ac8a52d7f1ba631017f1bc30f871be6, 8ac8a52d7f1ba631017f1bc30ff2274c, 8ac8a52d7f1ba631017f1bc3109037ad, 8ac8a52d7f1ba631017f1bc310a83ae5, 8ac8a52d7f1ba631017f1bc310172ab9, 8ac8a52d7f1ba631017f1bc30eb20783, 8ac8a52d7f1ba631017f1bc8da383c8c, 8ac8a52d7f1ba631017f1bc3109037b1, 8ac8a52d7f1ba631017f1bc3100629e9, 8ac8a52d7f1ba631017f1bc30f8f1caf, 8ac8a52d7f1ba631017f1bc310a73ac5, 8ac8a52d7f1ba631017f1bc3102a2c8c, 8ac8a52d7f1ba631017f1bc3108b36bf, 8ac8a52d7f1ba631017f1bc3100b2a44, 8ac8a52d7f1ba631017f1bc30f8f1cc9, 8ac8a52d7f1ba631017f1bc310ac3b65, 8ac8a52d7f1ba631017f1bc30f10100b, 8ac8a52d7f1ba631017f1bc30ef20d82, 8ac8a52d7f1ba631017f1bc310ad3b8e, 8ac8a52d7f1ba631017f1bc30f8f1cc8, 8ac8a52d7f1ba631017f1bc30f4c1577, 8ac8a52d7f1ba631017f1bc30e9b0650, 8ac8a52d7f1ba631017f1bc30ec50964, 8ac8a52d7f1ba631017f1bc30f8d1c94, 8ac8a52d7f1ba631017f1bc30eb507f6, 8ac8a52d7f1ba631017f1bc30f281251, 8ac8a52d7f1ba631017f1bc310a73ad5, 8ac8a52d7f1ba631017f1bc8da373c60, <snip> 2022-03-09 14:41:35,128 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.model.OwnerProductCurator - 293 owner-product relations removed 2022-03-09 14:41:35,175 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.model.OwnerProductCurator - 0 activation key product(s) removed 2022-03-09 14:41:35,454 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.controller.CandlepinPoolManager - Refresh pools for owner: redhat completed in: 2865ms 2022-03-09 14:41:35,615 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.controller.ManifestManager - Deleting stored manifest file: 8ac8a52d7f673a55017f6cfb689101bf 2022-03-09 14:41:35,714 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.async.tasks.ImportJob - Async import complete. 2022-03-09 14:41:35,794 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO org.candlepin.async.JobManager - Job "Import Manifest" completed in 5673ms -------------------------------------------------------------------------- # Before refreshing the manifest: su - postgres -c "psql candlepin -c \"select e.name, c.contenturl, c.vendor, c.label from cp2_environment_content ec left join cp_environment e on e.id = ec.environment_id left join cp2_content c on c.uuid = content_uuid order by e.name, c.vendor, c.label;\"" name | contenturl | vendor | label ---------------------------------+-----------------------------------------------------------------+---------+----------------------------------------- Library | /custom/hao_product/no-feed-repo | Custom | redhat_hao_product_no-feed-repo Library | /content/dist/rhel/server/7/$releasever/$basearch/os | Red Hat | rhel-7-server-rpms Library | /content/dist/rhel/server/7/7Server/$basearch/sat-tools/6.10/os | Red Hat | rhel-7-server-satellite-tools-6.10-rpms Library/cv-local-repos | /custom/hao_product/no-feed-repo | Custom | redhat_hao_product_no-feed-repo Library/sat-tools-only-imported | /content/dist/rhel/server/7/7Server/$basearch/sat-tools/6.10/os | Red Hat | rhel-7-server-satellite-tools-6.10-rpms # After refreshing the manifest. Only the custom environment contents are left: candlepin=# select e.name, c.contenturl, c.vendor, c.label from cp2_environment_content ec left join cp_environment e on e.id = ec.environment_id left join cp2_content c on c.uuid = content_uuid; name | contenturl | vendor | label ------------------------+----------------------------------+--------+--------------------------------- Library | /custom/hao_product/no-feed-repo | Custom | redhat_hao_product_no-feed-repo Library/cv-local-repos | /custom/hao_product/no-feed-repo | Custom | redhat_hao_product_no-feed-repo Adding the subscriptions back and refresh the manifest doesn't solve the issue. Expected results: Anything that we can improve the avoid this issue.