Bug 1600592

+++ This bug was initially created as a clone of Bug #1600201 +++

This is the Candlepin specific bug spawning from:

https://bugzilla.redhat.com/show_bug.cgi?id=1586210

"""
Description of problem:
Processing a virt-who report causes one specific RHSM request type is blocked for some time. Since these requests are fired frequently, this can cause the requests occupy whole passenger queue and passenger starts to return 503. 

Once the virt-who report processing is completed, the requests RHSM requests are unblocked. Anyway the 503 errors shouldnt happen meantime.


Version-Release number of selected component (if applicable):
Sat 6.3.1


How reproducible:
100% on customer data; generic reproducer shall not be hard to develop

Steps to Reproduce:
(generic reproducer)
1. Have few thousands of systems registered, with default certCheckInterval = 240 in rhsm.conf (the lower the better for reproducer)
2. Send virt-who report with mapping of several hundreds of systems
3. During processing of the report, check WebUI status or httpd error logs

Particular reproducer without a need of having a single host that is mimicked by specific curl requests:

A) to mimic RHSM certs check request: in fact just one particular URI GET request is essential / sufficient:

curl -s -u admin:changeme -X GET https://$(hostname -f)/rhsm/consumers/${uuid}/certificates/serials

(set uuid to various UUIDs of hosts / candlepin consumer IDs, and run these requests concurrently several times)

B) to mimic virt-who report: have virt-who-report.json with HV<->VMs mappings, and run:

time curl -s -u admin:changeme -X POST -H "Content-Type: application/json" -d @virt-who-report.json 'https://your.satellite/rhsm/hypervisors?owner=Owner&env=Library'



Actual results:
3. shows 503 errors in WebUI, /var/log/httpd/error_log having "Request queue is full. Returning an error" errors.


Expected results:
3. WebUI accessible, no such errors in httpd logs.


Additional info:
Technical explanation what goes wrong (to some extent):
- virt-who report processing requires updating katello_subscription_facets postgres table in some lengthy transaction (*)
- So Katello::Api::Rhsm::CandlepinProxiesController#serials requests are stuck on step:
 @host.subscription_facet.save!
for tens(!) of seconds, till the virt-who report is finished
- these requests come from the RHSM certs check queries / particular URI request /rhsm/consumers/${uuid}/certificates/serials
- these requests get accumulated for the few tens of seconds, and for higher load of them, this can fill whole passenger request queue
- that consequently triggers the 503 errors

Particular reproducer on customer data to be provided in next comment.
"""

--- Additional comment from Mike McCune on 2018-07-11 12:39:55 EDT ---

As noted here:


https://bugzilla.redhat.com/show_bug.cgi?id=1586210#c22

we are requesting an index be added to the cp_consumer_facts table (with a proper name) to speed up virt-who processing. We saw 30-40minute virt-who transactions reduced to ~5-6 minutes with the addition of this index.

"""
Ran the virt-who import and noticed that it is making a lot of relatively slow queries:

select cp_consumer.id from cp_consumer inner join cp_consumer_facts on cp_consumer.id = cp_consumer_facts.cp_consumer_id where cp_consumer_facts.mapkey = 'virt.uuid' and lower(cp_consumer_facts.element) in ( '?', '?') and cp_consumer.owner_id = '?' order by cp_consumer.updated desc

these were landing in the 200-300ms range.

I added an index:

echo "CREATE INDEX lower_case_test ON cp_consumer_facts ((lower(element)));" | sudo -u postgres psql -d candlepin


this dropped the above queries down into the 0.300ms range.

What this did was speed up the virt-who import from 30-45 minutes down to 6 minutes.

This helps overall performance, but we were still getting 503 errors during the 6 minute window of the virt-who import. I'd recommend creating this index in the short term to assist while we continue to investigate why the Satellite is unable to keep up with the load while virt-who import is running."""

--- Additional comment from Kevin Howell on 2018-07-12 10:45:49 EDT ---

Note, we'll need to evaluate in the context of recent changes made to handling of hypervisor checkins in https://github.com/candlepin/candlepin/pull/2035 . We may end up handling differently in candlepin-2.1, candlepin-2.4, and master as a result...