Hi Dirk, 

Thanks for opening the bug. With the given details we find it difficult to troubleshoot due to the custom facts in place. It will be helpful if you can provide us reproducer . 

thanks,
Rehana

Hi Rehana,

unfortunately it looks very random to me and I did not find a reproducer.
How about a possibility to regenerate the data in cp2_product_content instead of trying to find the cause of the missing data?

Kind Regards,
 Dirk

Dirk, Can you provide the "subscription-manager facts" & the generated entitlement certificate for one of these systems that is missing content? 

My suspicion is that The architecture the system is reporting doesn't match the repository. Arch is the one thing we filter on the Candlepin server.

Gentle reminder on the need info request above ( comment 3 ) .

I am unfortunately waiting to get the info from our customer myself.

A similar issue reported in the community forum (in this case CentOS 8 instead of SLES): https://community.theforeman.org/t/centos-8-subscriber-not-receiving-all-entitlements-via-subscription-manager/21187

There the architecture was confirmed as matching and there is also the output of rct.

Hello:

I am the OP of the forum post that Dirk linked to in the last update. The symptom is that the BaseOS repo is missing from the entitlement certificate issued to the consumer OS (CentOS 8 in this instance).

I have found in looking at the candlepin database that there is an arch set for BaseOS repository, but not for the AppStream or PowerTools repositories. I also found there is an entry missing in cp2_product_content for the BaseOS repo:

candlepin=# select * from cp2_content where name like '8%';
               uuid               |  content_id   |          created           |          updated           |          contenturl           |                       gpgurl          
              |                  label                   | metadataexpire |      name      | releasever | requiredtags | type | vendor | arches | entity_version | locked 
----------------------------------+---------------+----------------------------+----------------------------+-------------------------------+---------------------------------------
--------------+------------------------------------------+----------------+----------------+------------+--------------+------+--------+--------+----------------+--------
 8a9084ac754b9a7a017579eb9b611ff5 | 1603292218870 | 2020-10-21 10:56:58.869-04 | 2020-10-30 10:31:25.281-04 | /custom/CentOS/8_-_AppStream  | ../../katello/api/v2/repositories/3/gpg_key_content | My_Awesome_Org_CentOS_8_-_AppStream  |              1 | 8 - AppStream  |            |              | yum  | Custom |        |      497872048 |      0
 8a9084ac754b9a7a017579eb9b841ffa | 1603292227268 | 2020-10-21 10:57:07.265-04 | 2020-10-30 10:31:25.316-04 | /custom/CentOS/8_-_PowerTools | ../../katello/api/v2/repositories/5/gpg_key_content | My_Awesome_Org_CentOS_8_-_PowerTools |              1 | 8 - PowerTools |            |              | yum  | Custom |        |     1365813698 |      0
 8a9084ac754b9a7a01758f85c2a72445 | 1603292223082 | 2020-10-21 10:57:03.08-04  | 2020-11-03 14:11:49.415-05 | /custom/CentOS/8_-_BaseOS     | ../../katello/api/v2/repositories/4/gpg_key_content | My_Awesome_Org_CentOS_8_-_BaseOS     |              1 | 8 - BaseOS     |            |              | yum  | Custom | x86_64 |     1284365651 |      0
(3 rows)

candlepin=# select * from cp2_product_content where content_uuid='8a9084ac754b9a7a017579eb9b611ff5';
           product_uuid           |           content_uuid           | enabled |          created           |          updated           |                id                
----------------------------------+----------------------------------+---------+----------------------------+----------------------------+----------------------------------
 8a9084ac754b9a7a017579eb9b8f1ffc | 8a9084ac754b9a7a017579eb9b611ff5 | t       | 2020-10-30 10:31:25.327-04 | 2020-10-30 10:31:25.327-04 | 8a9084ac754b9a7a017579eb9b8f1fff
(1 row)

candlepin=# select * from cp2_product_content where content_uuid='8a9084ac754b9a7a017579eb9b841ffa';
           product_uuid           |           content_uuid           | enabled |          created           |          updated           |                id                
----------------------------------+----------------------------------+---------+----------------------------+----------------------------+----------------------------------
 8a9084ac754b9a7a017579eb9b8f1ffc | 8a9084ac754b9a7a017579eb9b841ffa | t       | 2020-10-30 10:31:25.327-04 | 2020-10-30 10:31:25.327-04 | 8a9084ac754b9a7a017579eb9b8f1ffd
(1 row)

candlepin=# select * from cp2_product_content where content_uuid='8a9084ac754b9a7a01758f85c2a72445';
 product_uuid | content_uuid | enabled | created | updated | id 
--------------+--------------+---------+---------+---------+----
(0 rows)

candlepin=#


Cheers,

Matthew

Hi , 

Thank you for the details, unfortunately we are unable to troubleshoot further with out clear reproducer. Can you please help us understand how the data was created ? 

thanks,
Rehana

I configured my environment using the hammer CLI utility.

I first created the product:

hammer product create --organization "My Awesome Org" --name "CentOS" --description "Community Enterprise Operating System" --sync-plan "Nightly Sync"

I then added each repository:

hammer repository create --organization "My Awesome Org" --content-type yum --download-policy immediate --http-proxy-policy global_default_http_proxy --publish-via-http false --product "CentOS" --name "8 - AppStream" --url "http://mirror.centos.org/centos-8/8/AppStream/x86_64/os" --ignorable-content "drpm,srpm,distribution"

hammer repository create --organization "My Awesome Org" --content-type yum --download-policy immediate --http-proxy-policy global_default_http_proxy --publish-via-http false --product "CentOS" --name "8 - BaseOS" --url "http://mirror.centos.org/centos-8/8/BaseOS/x86_64/os" --ignorable-content "drpm,srpm,distribution"

hammer repository create --organization "My Awesome Org" --content-type yum --download-policy immediate --http-proxy-policy global_default_http_proxy --publish-via-http false --product "CentOS" --name "8 - PowerTools" --url "http://mirror.centos.org/centos-8/8/PowerTools/x86_64/os" --ignorable-content "drpm,srpm,distribution"

I haven't manipulated Candlepin or its database in any other way.

Cheers,

Matthew

Hello,

One of my customers is also facing the same issue as per case 02826189.

Let me know if you need further details.

Is this issue reliably reproducible with other custom product additions? Are there any other details about the environment(s) that may have an impact on product/repo management? Any configuration on the organization or environment/library itself? Have there been other instances of this issue that you've noticed? Do you happen to have the (Candlepin) logs for when the custom product was added?

So far I've been unable to reproduce it locally, so while I put together something for a one-off recovery fix, I want to see if there are any other details here that I might be overlooking that's leading to this problem. I have one idea that could lead to this, but it's so far out in left field, I want to explore more grounded ideas first.

@crog I have been able to reproduce this issue with Foreman 1.24.3 and Foreman 2.1.4 on a RHEL 7 system. I am currently working with 2.1.4 as I assumed a newer version would resolve the issue. 2.1.4 is a clean, new environment. No data was migrated from the 1.24.3 environment in the event there was corrupt data causing the issue.

I unfortunately don't have Candlepin logs from when the custom Product and Repos were added (this system was built late Oct 2020). I will need to re-build this system so I can capture the logging data requested. I do have Candlepin logs going back to late Nov 2020, but the only errors are the two messages below which appear from time-to-time:

2020-12-16 00:00:00,029 [thread=Thread-3377 (ActiveMQ-client-global-threads)] [=, org=, csid=] ERROR org.candlepin.async.JobManager - Unable to find job status for message: JobMessage [id: ActiveEntitlementJob, key: 8a9084ac754b9a7a017669eb58812ecd]

2020-12-16 00:00:00,030 [thread=Thread-3377 (ActiveMQ-client-global-threads)] [=, org=, csid=] ERROR org.candlepin.async.JobMessageReceiver - Job processing failed terminally; committing job message as acknowledged: org.candlepin.messaging.impl.artemis.ArtemisMessage@49d77fff

2020-12-16 19:00:00,024 [thread=Thread-3420 (ActiveMQ-client-global-threads)] [=, org=, csid=] ERROR org.candlepin.async.JobManager - Unable to find job status for message: JobMessage [id: ActiveEntitlementJob, key: 8a9084ac754b9a7a01766dff0c012efa]

2020-12-16 19:00:00,024 [thread=Thread-3420 (ActiveMQ-client-global-threads)] [=, org=, csid=] ERROR org.candlepin.async.JobMessageReceiver - Job processing failed terminally; committing job message as acknowledged: org.candlepin.messaging.impl.artemis.ArtemisMessage@6848d8e3

2020-12-16 00:00:00,029 [thread=Thread-3377 (ActiveMQ-client-global-threads)] [=, org=, csid=] ERROR org.candlepin.async.JobManager - Unable to find job status for message: JobMessage [id: ActiveEntitlementJob, key: 8a9084ac754b9a7a017669eb58812ecd]

2020-12-16 00:00:00,030 [thread=Thread-3377 (ActiveMQ-client-global-threads)] [=, org=, csid=] ERROR org.candlepin.async.JobMessageReceiver - Job processing failed terminally; committing job message as acknowledged: org.candlepin.messaging.impl.artemis.ArtemisMessage@49d77fff

2020-12-16 19:00:00,024 [thread=Thread-3420 (ActiveMQ-client-global-threads)] [=, org=, csid=] ERROR org.candlepin.async.JobManager - Unable to find job status for message: JobMessage [id: ActiveEntitlementJob, key: 8a9084ac754b9a7a01766dff0c012efa]

2020-12-16 19:00:00,024 [thread=Thread-3420 (ActiveMQ-client-global-threads)] [=, org=, csid=] ERROR org.candlepin.async.JobMessageReceiver - Job processing failed terminally; committing job message as acknowledged: org.candlepin.messaging.impl.artemis.ArtemisMessage@6848d8e3

2020-12-18 22:00:00,022 [thread=Thread-3535 (ActiveMQ-client-global-threads)] [=, org=, csid=] ERROR org.candlepin.async.JobManager - Unable to find job status for message: JobMessage [id: ExpiredPoolsCleanupJob, key: 8a9084ac754b9a7a017678f08f822f6b]

2020-12-18 22:00:00,023 [thread=Thread-3535 (ActiveMQ-client-global-threads)] [=, org=, csid=] ERROR org.candlepin.async.JobMessageReceiver - Job processing failed terminally; committing job message as acknowledged: org.candlepin.messaging.impl.artemis.ArtemisMessage@3926be7c

2020-12-18 22:00:00,022 [thread=Thread-3535 (ActiveMQ-client-global-threads)] [=, org=, csid=] ERROR org.candlepin.async.JobManager - Unable to find job status for message: JobMessage [id: ExpiredPoolsCleanupJob, key: 8a9084ac754b9a7a017678f08f822f6b]

2020-12-18 22:00:00,023 [thread=Thread-3535 (ActiveMQ-client-global-threads)] [=, org=, csid=] ERROR org.candlepin.async.JobMessageReceiver - Job processing failed terminally; committing job message as acknowledged: org.candlepin.messaging.impl.artemis.ArtemisMessage@3926be7c

2020-12-22 00:00:00,037 [thread=Thread-3728 (ActiveMQ-client-global-threads)] [=, org=, csid=] ERROR org.candlepin.async.JobManager - Unable to find job status for message: JobMessage [id: ActiveEntitlementJob, key: 8a9084ac754b9a7a017688d180813012]

2020-12-22 00:00:00,037 [thread=Thread-3728 (ActiveMQ-client-global-threads)] [=, org=, csid=] ERROR org.candlepin.async.JobMessageReceiver - Job processing failed terminally; committing job message as acknowledged: org.candlepin.messaging.impl.artemis.ArtemisMessage@310bc644

2020-12-22 00:00:00,037 [thread=Thread-3728 (ActiveMQ-client-global-threads)] [=, org=, csid=] ERROR org.candlepin.async.JobManager - Unable to find job status for message: JobMessage [id: ActiveEntitlementJob, key: 8a9084ac754b9a7a017688d180813012]

2020-12-22 00:00:00,037 [thread=Thread-3728 (ActiveMQ-client-global-threads)] [=, org=, csid=] ERROR org.candlepin.async.JobMessageReceiver - Job processing failed terminally; committing job message as acknowledged: org.candlepin.messaging.impl.artemis.ArtemisMessage@310bc644

I will post an update once I have rebuilt the system and can provide additional Candlepin logs.

Alright. I'll retry my testing with 2.1.4 and see if I can get the same disconnect.

Couple notes for checking the state of things:
- Entities originating from cp2_products and cp2_content are considered to be immutable. So whenever one of them changes, a new entity is created and the old ones get cleaned up eventually. That said, when checking the UUIDs from the CP database, you have to follow the chain from cp_pool.product_uuid.

- Assuming the content is supposed to be linked to a specific product, adding the missing entries in the join table shouldn't be an issue, but is real tedious.

- The arch coming out of nowhere is something that's standing out to me. If the client is being registered with an arch other than x86_64, you'll get the behavior you're seeing, even with the proper linkage in the database. So that'd be something to investigate.



As far as the job messages you're seeing, those are (likely) unrelated to this issue. That'll happen if Artemis delivers a job message for a job that no longer exists in the database. I've seen this happen in cases where the database is erroneously truncated or otherwise modified such that jobs are removed from cp_async_jobs after a message is dispatched, but before it is delivered and processed. It *should* only happen once for every such message, as Candlepin should be ACKing them on a terminal failure. If you keep seeing these, it might be worth opening another BZ to start an investigation on that.

I'll keep an eye out for the next update to see if the Candlepin logs have anything illuminating.

An update from my end. Try as I might, following the repro steps above, I still don't get this into the state observed. The commands issued:


hammer organization create --name "test_org"

hammer sync-plan create --organization test_org --name "Nightly Sync" --enabled true --sync-date "2021-01-30 00:00:00" --interval "daily"

hammer product create --organization "test_org" --name "CentOS" --description "Community Enterprise Operating System" --sync-plan "Nightly Sync"

hammer repository create --organization "test_org" --content-type yum --download-policy immediate --http-proxy-policy global_default_http_proxy --publish-via-http false --product "CentOS" --name "8 - AppStream" --url "http://mirror.centos.org/centos-8/8/AppStream/x86_64/os" --ignorable-content "drpm,srpm,distribution"

hammer repository create --organization "test_org" --content-type yum --download-policy immediate --http-proxy-policy global_default_http_proxy --publish-via-http false --product "CentOS" --name "8 - BaseOS" --url "http://mirror.centos.org/centos-8/8/BaseOS/x86_64/os" --ignorable-content "drpm,srpm,distribution"

hammer repository create --organization "test_org" --content-type yum --download-policy immediate --http-proxy-policy global_default_http_proxy --publish-via-http false --product "CentOS" --name "8 - PowerTools" --url "http://mirror.centos.org/centos-8/8/PowerTools/x86_64/os" --ignorable-content 




SELECT pool.id AS "Pool ID", pool.product_uuid as "Product UUID", prod.name as "Product Name", pc.content_uuid as "Content UUID", cont.name as "Content Name", cont.label AS "Content Label" FROM cp_pool pool JOIN cp2_products prod ON pool.product_uuid = prod.uuid JOIN cp2_product_content pc ON prod.uuid = pc.product_uuid JOIN cp2_content cont ON pc.content_uuid = cont.uuid;

             Pool ID              |           Product UUID           | Product Name |           Content UUID           |  Content Name  |         Content Label          
----------------------------------+----------------------------------+--------------+----------------------------------+----------------+--------------------------------
 ff80808176f319710176f744050c0030 | ff80808176f319710176f7463f51003d | CentOS       | ff80808176f319710176f744dffd0031 | 8 - AppStream  | test_org_CentOS_8_-_AppStream
 ff80808176f319710176f744050c0030 | ff80808176f319710176f7463f51003d | CentOS       | ff80808176f319710176f745ce3b0036 | 8 - BaseOS     | test_org_CentOS_8_-_BaseOS
 ff80808176f319710176f744050c0030 | ff80808176f319710176f7463f51003d | CentOS       | ff80808176f319710176f7463f0a003c | 8 - PowerTools | test_org_CentOS_8_-_PowerTools
(3 rows)



Are there any other steps or configuration that are missing from my steps that could be contributing to the issue here?

@crog 

I have a bittersweet update; it appears a fresh install has "resolved" the issue. As such, I cannot provide the logs requested and I'm unable to reproduce the issue with a subsequent "wipe and re-install." I have tried to keep things in a "change-freeze" state, so package versions have not changed. I cannot find anything else that has changed. I am going to continue monitoring it and will report back if it rears its head again.

Well, that's good-ish news, I guess. These are always the most frustrating bugs to deal with.

If you still have your affected environment around, the SQL query I provided above should give you the same output I see (as well as that for several other products -- feel free to add another condition such that you only view the appropriate pool/product). If it doesn't, the safest thing to do would probably be to delete the problematic repo entirely, and then re-add it using hammer commands.

The other thing to watch out for is the arches. In the original post, the arch on the BaseOS repo was "x86_64" (which is odd since the command didn't include an arch), but if the client receiving the entitlement declares itself as using any other arch, the entitlement cert's content would be missing it. The fix there would be to update the repo to no longer have the arch if it is there erroneously. There's an undocumented way to do it through hammer if you modify one file in katello (basically add recognition for the --arch param), or you can manually update the content within the Candlepin DB and restart it. The latter comes with a bunch of extra churn due to the entity version no longer being valid, but it should* work otherwise. I would strongly recommend the former, however.

Dirk, 

Have you received any updates from your customer on this issue?

Unfortunately they have completely worked around the issue by skipping subscription-manager and directly assigning the repositories from the published URL.
SLES also makes deletion and recreation until everything works impossible as the SCC manager plugin does only allow to add a product and if you delete a repository directly there is not re-adding. 
Also SLES repositories are bloated with other archs, debuginfo and similar, what was the reason to limit it to x86_64 if I remember correctly.

Kshitij, any updates on your end here?

Lacking a proper reproducer or information to pursue on the matter, I'm closing this as "cannot reproduce."

If anyone has any further information on the issue or has any details that could lead to a reproducer, please reopen the BZ and we'll make another attempt at identifying and fixing this problem.

Hello,

This is been tracked in a new Bugzilla so yes this should be closed now as a duplicate.