Bug 2064979 - Clients can't subscribe to or enable Red Hat repositories after renewing subscriptions
Summary: Clients can't subscribe to or enable Red Hat repositories after renewing subs...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Candlepin
Version: 6.10.3
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: 6.12.0
Assignee: satellite6-bugs
QA Contact:
URL:
Whiteboard:
Depends On: 2068122 2069160 2069165
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-03-17 02:46 UTC by Hao Chang Yu
Modified: 2024-03-27 13:54 UTC (History)
19 users (show)

Fixed In Version: candlepin-4.0.19-1, candlepin-4.1.14-1, candlepin-4.2.3-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2068122 2069160 2069165 2112400 (view as bug list)
Environment:
Last Closed: 2022-11-16 13:33:45 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker ENT-4854 0 None None None 2022-03-28 12:13:09 UTC
Red Hat Issue Tracker SAT-11410 0 None None None 2022-08-30 18:32:18 UTC
Red Hat Knowledge Base (Solution) 6997845 0 None None None 2023-02-15 14:18:32 UTC
Red Hat Product Errata RHSA-2022:8506 0 None None None 2022-11-16 13:34:01 UTC

Description Hao Chang Yu 2022-03-17 02:46:13 UTC
Description of problem:
If user performs a manifest refresh against a manifest having all expired subscriptions or an empty manifest, Candlepin will remove all the environment contents in the organization. After renewing all subscriptions and attach to the manifest, user performs a manifest refresh again. The job completed successfully but all clients are not able to subscribe to or enable any Redhat repositories due to no content listing.


Steps to Reproduce:
1. Have a valid manifest imported to the Satellite. Enable and sync some Redhat repositories. Create some custom repositories. Create one or more content views and attach some repositories.
2. Go to Web UI -> Content ->  Subscriptions -> Manage Manifest -> click the "Subscription Allocation" link to go to the Customer Portal page.
3. In the Customer Portal, delete all subscriptions allocated to the manifest.
4. Then refresh the manifest using the Satellite Web UI.

Actual results:
# You will see the import job remove all products and contents references like below in candlepin.log file.

NOTE: If you don't see the "Removing owner-content references for owner" and "Removing owner-product references for owner", you will need to follow step (4) again. I refreshed the empty manifest twice to reproduce the issue. Not sure why.

--------------------------------------------------------------------------
2022-03-09 14:41:29,987 [thread=http-bio-127.0.0.1-23443-exec-3] [req=67b1d364-ddd4-4c72-a2f6-09291d658b8d, org=, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/owners/redhat/imports/async
2022-03-09 14:41:30,000 [thread=http-bio-127.0.0.1-23443-exec-3] [req=67b1d364-ddd4-4c72-a2f6-09291d658b8d, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO  org.candlepin.resource.OwnerResource - Running async import of archive /var/cache/tomcat/temp/pfx11102612137104972728sfx for owner redhat
2022-03-09 14:41:30,036 [thread=http-bio-127.0.0.1-23443-exec-3] [req=67b1d364-ddd4-4c72-a2f6-09291d658b8d, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO  org.candlepin.async.JobManager - Job queued: AsyncJobStatus [id: 8ac8a52d7f673a55017f6cfb68ad01c0, name: Import Manifest, key: ImportJob, state: QUEUED]
2022-03-09 14:41:30,046 [thread=http-bio-127.0.0.1-23443-exec-3] [req=67b1d364-ddd4-4c72-a2f6-09291d658b8d, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO  org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=60
2022-03-09 14:41:30,063 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO  org.candlepin.async.JobManager - Starting job "Import Manifest" using class: org.candlepin.async.tasks.ImportJob
2022-03-09 14:41:30,101 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO  org.candlepin.sync.RulesImporter - Importing new rules from manifest, current version: 5.41 new version: 5.41
2022-03-09 14:41:30,129 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO  org.candlepin.policy.js.JsRunnerProvider - Recompiling rules with timestamp: 2022-03-09 14:41:30.103
2022-03-09 14:41:32,583 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] WARN  org.candlepin.sync.Importer - No products found to import, skipping product import.
2022-03-09 14:41:32,585 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] WARN  org.candlepin.sync.Importer - No entitlements in manifest, removing all subscriptions for owner.
2022-03-09 14:41:32,589 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO  org.candlepin.controller.CandlepinPoolManager - Refreshing pools for owner: Owner [id: 8ac8a52d7f1ba631017f1bb6f3ed0001, key: redhat]
2022-03-09 14:41:32,851 [thread=http-bio-127.0.0.1-23443-exec-7] [req=634aef21-933c-4cd6-adee-466ae61108c4, org=, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=GET, uri=/candlepin/jobs/8ac8a52d7f673a55017f6cfb68ad01c0?
2022-03-09 14:41:32,856 [thread=http-bio-127.0.0.1-23443-exec-7] [req=634aef21-933c-4cd6-adee-466ae61108c4, org=, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO  org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=5
2022-03-09 14:41:33,641 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO  org.candlepin.model.OwnerContentCurator - Removing owner-content references for owner: Owner [id: 8ac8a52d7f1ba631017f1bb6f3ed0001, key: redhat], [8ac8a52d7f1ba631017f1bc30ef50dbf, 8ac8a52d7f1ba631017f1bc30ef50dbb, 8ac8a52d7f1ba631017f1bc30ef50dbc, 8ac8a52d7f1ba631017f1bc30ef50dbd, 8ac8a52d7f1ba631017f1bc30ef50dbe, 8ac8a52d7f1ba631017f1bc30ef50dca, 8ac8a52d7f1ba631017f1bc310653139, 8ac8a52d7f1ba631017f1bc310653138, 8ac8a52d7f1ba631017f1bc310653137, 8ac8a52d7f1ba631017f1bc310653136, 8ac8a52d7f673a55017f675abd700080, 8ac8a52d7f1ba631017f1bc310653135, 8ac8a52d7f1ba631017f1bc310653134, 8ac8a52d7f1ba631017f1bc30e77045f, 8ac8a52d7f1ba631017f1bc31065313b, 8ac8a52d7f1ba631017f1bc31065313a, 8ac8a52d7f1ba631017f1bc30ffc28a1, 8ac8a52d7f1ba631017f1bc30ffc28a0, 8ac8a52d7f1ba631017f1bc31065313f, 8ac8a52d7f1ba631017f1bc31065313e, 8ac8a52d7f1ba631017f1bc31065313d, 8ac8a52d7f1ba631017f1bc31065313c, 8ac8a52d7f1ba631017f1bc30e770465, 8ac8a52d7f1ba631017f1bc30e770466, 8ac8a52d7f1ba631017f1bc30ef50db9, 8ac8a52d7f1ba631017f1bc310653140, 8ac8a52d7f1ba631017f1bc30ef50dc0, 8ac8a52d7f1ba631017f1bc30ef50dc1, 8ac8a52d7f1ba631017f1bc3102d2ce7, 8ac8a52d7f1ba631017f1bc30ef50dc2, 8ac8a52d7f1ba631017f1bc3102d2ce5, 8ac8a52d7f1ba631017f1bc3102d2ce6, 8ac8a52d7f1ba631017f1bc3102d2ce3, 8ac8a52d7f1ba631017f1bc3102d2ce4, 8ac8a52d7f1ba631017f1bc30e770461, 8ac8a52d7f1ba631017f1bc3102d2ce1, 8ac8a52d7f1ba631017f1bc30e770462, 8ac8a52d7f1ba631017f1bc3102d2ce2, 8ac8a52d7f1ba631017f1bc30e770463, 8ac8a52d7f1ba631017f1bc3102d2ce0, 8ac8a52d7f1ba631017f1bc30e770464, 8ac8a52d7f1ba631017f1bc30e770460, 8ac8a52d7f673a55017f675abcd0000a, 8ac8a52d7f1ba631017f1bc30ef50dba, 8ac8a52d7f1ba631017f1bc3102d2cca, 8ac8a52d7f1ba631017f1bc30ef50dc7, 8ac8a52d7f1ba631017f1bc30ef50dc8, 8ac8a52d7f1ba631017f1bc30ef50dc9, 8ac8a52d7f1ba631017f1bc30ef50dc3, 8ac8a52d7f1ba631017f1bc30ef50dc4, 8ac8a52d7f1ba631017f1bc30ef50dc5, 8ac8a52d7f1ba631017f1bc30ef50dc6, 8ac8a52d7f1ba631017f1bc30eb80829, 8ac8a52d7f1ba631017f1bc30eb80827, 8ac8a52d7f1ba631017f1bc30eb80828, 8ac8a52d7f1ba631017f1bc30eb8081a,
<snip>
2022-03-09 14:41:35,076 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO  org.candlepin.model.OwnerProductCurator - Removing owner-product references for owner: Owner [id: 8ac8a52d7f1ba631017f1bb6f3ed0001, key: redhat], [8ac8a52d7f1ba631017f1bc310402ef8, 8ac8a52d7f1ba631017f1bc3108f3782, 8ac8a52d7f1ba631017f1bc30ede0bb3, 8ac8a52d7f1ba631017f1bc30e730431, 8ac8a52d7f1ba631017f1bc3107e34eb, 8ac8a52d7f1ba631017f1bc310a039bd, 8ac8a52d7f1ba631017f1bc30ef80e14, 8ac8a52d7f1ba631017f1bc30eef0d29, 8ac8a52d7f1ba631017f1bc30ee40c42, 8ac8a52d7f1ba631017f1bc30f3a143f, 8ac8a52d7f1ba631017f1bc30ecd0a1a, 8ac8a52d7f1ba631017f1bc3102a2c79, 8ac8a52d7f1ba631017f1bc3108e3742, 8ac8a52d7f1ba631017f1bc310a83af3, 8ac8a52d7f1ba631017f1bc30ed00a67, 8ac8a52d7f1ba631017f1bc30f871be6, 8ac8a52d7f1ba631017f1bc30ff2274c, 8ac8a52d7f1ba631017f1bc3109037ad, 8ac8a52d7f1ba631017f1bc310a83ae5, 8ac8a52d7f1ba631017f1bc310172ab9, 8ac8a52d7f1ba631017f1bc30eb20783, 8ac8a52d7f1ba631017f1bc8da383c8c, 8ac8a52d7f1ba631017f1bc3109037b1, 8ac8a52d7f1ba631017f1bc3100629e9, 8ac8a52d7f1ba631017f1bc30f8f1caf, 8ac8a52d7f1ba631017f1bc310a73ac5, 8ac8a52d7f1ba631017f1bc3102a2c8c, 8ac8a52d7f1ba631017f1bc3108b36bf, 8ac8a52d7f1ba631017f1bc3100b2a44, 8ac8a52d7f1ba631017f1bc30f8f1cc9, 8ac8a52d7f1ba631017f1bc310ac3b65, 8ac8a52d7f1ba631017f1bc30f10100b, 8ac8a52d7f1ba631017f1bc30ef20d82, 8ac8a52d7f1ba631017f1bc310ad3b8e, 8ac8a52d7f1ba631017f1bc30f8f1cc8, 8ac8a52d7f1ba631017f1bc30f4c1577, 8ac8a52d7f1ba631017f1bc30e9b0650, 8ac8a52d7f1ba631017f1bc30ec50964, 8ac8a52d7f1ba631017f1bc30f8d1c94, 8ac8a52d7f1ba631017f1bc30eb507f6, 8ac8a52d7f1ba631017f1bc30f281251, 8ac8a52d7f1ba631017f1bc310a73ad5, 8ac8a52d7f1ba631017f1bc8da373c60,
<snip>
2022-03-09 14:41:35,128 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO  org.candlepin.model.OwnerProductCurator - 293 owner-product relations removed
2022-03-09 14:41:35,175 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO  org.candlepin.model.OwnerProductCurator - 0 activation key product(s) removed
2022-03-09 14:41:35,454 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO  org.candlepin.controller.CandlepinPoolManager - Refresh pools for owner: redhat completed in: 2865ms
2022-03-09 14:41:35,615 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO  org.candlepin.controller.ManifestManager - Deleting stored manifest file: 8ac8a52d7f673a55017f6cfb689101bf
2022-03-09 14:41:35,714 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO  org.candlepin.async.tasks.ImportJob - Async import complete.
2022-03-09 14:41:35,794 [thread=Thread-75 (ActiveMQ-client-global-threads)] [job=8ac8a52d7f673a55017f6cfb68ad01c0, job_key=ImportJob, org=redhat, csid=90346f37-caa9-49d6-8cff-7a9ff49f85c3] INFO  org.candlepin.async.JobManager - Job "Import Manifest" completed in 5673ms
--------------------------------------------------------------------------


# Before refreshing the manifest:

su - postgres -c "psql candlepin -c \"select e.name, c.contenturl, c.vendor, c.label  from cp2_environment_content ec left join cp_environment e on e.id = ec.environment_id left join cp2_content c on c.uuid = content_uuid order by e.name, c.vendor, c.label;\""
              name               |                           contenturl                            | vendor  |                  label                  
---------------------------------+-----------------------------------------------------------------+---------+-----------------------------------------
 Library                         | /custom/hao_product/no-feed-repo                                | Custom  | redhat_hao_product_no-feed-repo
 Library                         | /content/dist/rhel/server/7/$releasever/$basearch/os            | Red Hat | rhel-7-server-rpms
 Library                         | /content/dist/rhel/server/7/7Server/$basearch/sat-tools/6.10/os | Red Hat | rhel-7-server-satellite-tools-6.10-rpms
 Library/cv-local-repos          | /custom/hao_product/no-feed-repo                                | Custom  | redhat_hao_product_no-feed-repo
 Library/sat-tools-only-imported | /content/dist/rhel/server/7/7Server/$basearch/sat-tools/6.10/os | Red Hat | rhel-7-server-satellite-tools-6.10-rpms


# After refreshing the manifest. Only the custom environment contents are left:

candlepin=# select e.name, c.contenturl, c.vendor, c.label  from cp2_environment_content ec left join cp_environment e on e.id = ec.environment_id left join cp2_content c on c.uuid = content_uuid;
          name          |            contenturl            | vendor |              label              
------------------------+----------------------------------+--------+---------------------------------
 Library                | /custom/hao_product/no-feed-repo | Custom | redhat_hao_product_no-feed-repo
 Library/cv-local-repos | /custom/hao_product/no-feed-repo | Custom | redhat_hao_product_no-feed-repo



Adding the subscriptions back and refresh the manifest doesn't solve the issue.


Expected results:
Anything that we can improve the avoid this issue.

Comment 5 Hao Chang Yu 2022-03-18 02:13:33 UTC
The workaround of this issue is:

Steps:
1) Complete resync all Redhat repositories.
2) Publish new version and promote all content views.
3) Publish new version and promote all composite content views if any.

Comment 24 Lai 2022-08-01 20:26:52 UTC
Steps to Retest:
1. Have a valid manifest imported to the Satellite. Enable and sync some Redhat repositories. Create some custom repositories. Create one or more content views and attach some repositories.
2. Go to Web UI -> Content ->  Subscriptions -> Manage Manifest -> click the "Subscription Allocation" link to go to the Customer Portal page.
3. In the Customer Portal, delete all subscriptions allocated to the manifest.
4. Then refresh the manifest using the Satellite Web UI.
5. Run grep for "Removing owner-content references for owner" on /var/log/candlepin/candlepin.log
6. Run su - postgres -c "psql candlepin -c \"select e.name, c.contenturl, c.vendor, c.label  from cp2_environment_content ec left join cp_environment e on e.id = ec.environment_id left join cp2_content c on c.uuid = content_uuid order by e.name, c.vendor, c.label;\""

Expected result:
5. Should return empty
6. Should return only the number of repos that was synced and added to cv

Actual:
5. Did return empty
6. Did return only the number of repos that was synced and added to cv

I also did manifest refresh a few times to ensure the error didn't occur anymore.  Everything seems to check out ok.

Verified on 6.12 snap 4

Comment 29 errata-xmlrpc 2022-11-16 13:33:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.12 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8506


Note You need to log in before you can comment on or make changes to this bug.