Bug 1510136

Description of problem:
Currently, there are several cases popping up around RHUI entitlement certs being suddenly invalid, resulting in a lack of ability to successfully sync new content from the CDN.

This appears to be resulting from a chain of events which are normal for almost all systems in respect for this, but is unaccounted for in RHUI.

At this point, Candlepin 2 will mark certificates dirty when they need to be updated. This could be for various reason, including updates to product content (paths, or otherwise), new engIDs, etc..

When Candlepin 2 marks this certificate dirty, this tells subscription-manager that during the next check-in with the customer portal, the system should generate and receive updated entitlement certificates with the new/corrected information.

Right now, we are also experiencing the process of refreshing all accounts to accommodate some missing information at the point where Candlepin 2 was rolled out to production. This means that same product data was unavailable or incorrect, and is being pushed out to all the of the accounts that are being refreshed.

With this information in mind, the issue is currently more widespread than would normally be experienced, but none-the-less needs to be accounted for in RHUI.

Essentially, my current understanding is as follows:

If your RHUI server is actively registered and subscribed (itself) to the customer portal via subscription-manager, and is utilizing the same consumer/distributor upstream as is being used to download the entitlement certificate that is imported into `rhui-manager`, if any content on that entitlement cert changes upstream (regardless of action via customer), it will be marked for regeneration - which will occur during the next subscription-manager check-in, rendering the currently imported entitlement cert within `rhui-manager` to be obsolete. This will result in the lack of ability to continue to sync CDN content into the RHUI environment.

By my understanding, this would affect RHUI 2 and 3.

One idea for updates regarding this issue would be for RHUI to make use of entitlement certificates that are present in /etc/pki/entitlement, rather than expected manual user intervention to update the certificates themselves via `rhui-manager`.

Version-Release number of selected component (if applicable):
Candlepin 2 Prod
RHUI 2, 3

How reproducible:
100% given right conditions: 

Steps to Reproduce:
1. Need an out of date entitlement cert with missing/incorrect content, and then an account refresh.
2. RHUI could stop successful syncs at this point of the entitlement cert was marked dirty, and sub-man check-in refreshes the data.

Actual results:
RHUI can no longer sync until a new entitlement certificate is uploaded into rhui-manager. (can be obtained from customer portal or /etc/pki/entitlement)

Expected results:
RHUI needs a (documented) method of utilizing a different entitlement cert that would not be touched by a subscription-manager check-in, or should make use of the entitlement certs that would be present on the system after a refresh.

Additional info:

This information is as I remember it being described and could be slightly off in various places - further information and requests for technical data should be brought to bcourt@redhat on the Candlepin side.