Bug 1928837

Description of problem:

Content Hosts are receiving an entitlement certificate that lacks the "Content" section. As such, entitled hosts are unable to access the product's repository. Currently, this is affecting only RHEL 8 hosts that consume a custom product (Icinga2). I have tried to call /candlepin/entitlements/product/275076129832?lazy_regen=true and /rhsm/consumers/36a1ad9e-f4d8-494a-959a-d795079ee775/certificates?lazy_regen=true, but to no avail. In previous instances, this has affected RHEL 7 and CentOS 8. At the moment, though, only RHEL 8 Content Hosts.

Version-Release number of selected component (if applicable):

$ rpm -qi candlepin
Name        : candlepin
Version     : 3.1.16
Release     : 1.el7
Architecture: noarch
Install Date: Fri 15 Jan 2021 11:28:06 AM EST
Group       : System Environment/Daemons
Size        : 71449587
License     : GPLv2
Signature   : RSA/SHA256, Tue 14 Jul 2020 10:19:52 AM EDT, Key ID 628005a4b6f08ccf
Source RPM  : candlepin-3.1.16-1.el7.src.rpm
Build Date  : Thu 09 Jul 2020 03:21:28 PM EDT
Build Host  : koji.katello.org
Relocations : (not relocatable)
Packager    : Koji
Vendor      : Koji
URL         : http://www.candlepinproject.org
Summary     : Candlepin is an open source entitlement management system
Description : Candlepin is an open source entitlement management system.

How reproducible:

At first this appeared to be an anomaly that occurred during install. However, this issue has (after a wipe and re-install) reared its head after registering 100+ hosts. We attempted to debug the issue on BZ# 1886325, but to no avail. Fortunately, the issue started soon enough that I've been able to backup all logs from the time of the wipe+reinstall and now.

Steps to Reproduce:

1. Install Foreman (foreman-2.1.4-1) + Katello (katello-3.16.2-1)
2. Register some hosts and consume RHEL + Custom products
3. Validate access to repos
4. After some time, register a lot of hosts
5. Run `subscription-manager refresh` and validate all hosts have access to Custom product repos

Actual results:

Albeit entitled via Activation Key, or subscribing manually via `subscription-manager`, some hosts will inexplicably loose access to a repository.

Expected results:

If a Content Host is subscribed to a product, all applicable repositories should be included in the product entitlement certificate (i.e. the Content section).

Additional info:

Since steps to reproduce are unknown (exactly), I am willing to work directly with Red Hat and grant remote access to my system for further troubleshooting. Whether this is truly a bug or documentation issue, I'd like to do whatever I can to help get it resolved.