Red Hat Bugzilla – Bug 1431783
Stop calling back to candlepin for qpid messages
Last modified: 2017-08-10 13:02:29 EDT
Description of problem: Currently candlepin is sending large messages through candlepin and we are taking these messages, pulling out some ID and fetching the data from candlepin. All the data we need is right there. We should just use it directly. This should speed up processing of events considerably.
Created redmine issue http://projects.theforeman.org/issues/19026 from this bug
Do I understand this BZ right, that currently: For some events sent from candlepin to katello via katello_event_queue, katello contacts candlepin to do some redundant update/query? Since I was testing few types of events sent from candlepin->katello and no candlepin request was generated for either of them. Is one such event type "compliance.created" ? And if katello tries to contact candlepin but candlepin is down, does it raise "A backend service [ Candlepin ] is unreachable" error and is ListenOnCandlepinEvents task paused in error state? Will a fix of this BZ help also to this behaviour (temporary candlepin down pauses L.O.C.E. task forever, causing the katello_event_queue backlog grows over time)?
Upstream bug assigned to jsherril@redhat.com
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/19026 has been resolved.
Pavel, I'm not aware of that issue, but yes if it is an issue it should resolve it. Currently we look for: entitlement.create entitlement.deleted pool.created pool.deleted compliance.created Prior to this BZ's fix we were going to candlepin to fetch either pool info, or consumer info for every single one these messages. An expiring manifest could cause 10,000 - 20,000 of these messages to flood in. Now we've made a couple of changes: compliance.created does not contact candlepin at all. The other 4 types are cause 'events' to be put on the katello event queue (an in database queue that is processed by the "Katello Event Monitor" task. This task should features some de-duplication built in, so that 10,000 entitlement.create or delete messages on the same pool would only result in a handful of calls to candlepin.
Adding 2nd upstream issue and moving to POST
There are several scenarios that need to be tested for a) correct functionality and b) that extra calls aren't making it to candlepin Scenario A) 1. make sure you have a manifest imported 2. Register a rhel content host 3. run subscription-manager attach --auto 4. check subscription status and verify it should be green 5. run subscription-manager remove --all 6. check subscription status and verify it should be red Scenario B) 1. make sure you have a manifest imported 2. Register a rhel content host 3. Attach a redhat subscription via the UI 4. check subscription status and verify it should be green 5. Remove the subscription from the UI 6. check subscription status and verify it should be red Scenario C) 1. Import a manifest, verify that all subscriptions and pools show up properly on the Red Hat Subscriptions Page Scenario D) 1. Import a manifest with at least 50-100 Subs 2. Register at least 50-100 systems, and have them attach the subs from the manifest. These do not need to be 'real' systems 3. Monitor /var/log/candlepin/candlepin.log and look for "Request" 4. On 6.2.10 you likely would see at least 200 requests (2 per system) hitting candlepin. (I would try this on a 6.2.10 box so that you are confident in the behavior). 5. On 6.2.11 you should not see 2 requests per system. You may see a handful (especially around the deletion, but nowhere near 2 per system).
Created attachment 1305977 [details] Green content host status subscription-manager attach --auto
Created attachment 1305978 [details] Red content host status subscription-manager remove --all
Created attachment 1305979 [details] Red Hat Subscriptions screenshot
Created attachment 1305997 [details] Green status from UI Attached subscriptions through web UI
Created attachment 1305998 [details] FAILED- Red status from UI After deleting RH subscriptions using the web UI the content host status was green but expected red.
Created attachment 1305999 [details] Red status from UI Deleting subscriptions from content host changed status to red.
-bash-4.2# grep -c "Request" /var/log/candlepin/candlepin.log 242 -bash-4.2# grep -c "Request" /var/log/candlepin/candlepin.log 134
Verified in Satellite 6.2.11 Snap 2
*** Bug 1414159 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2466