Red Hat Bugzilla – Bug 869286
SCALE: Subscription of systems gets slower and slower as number of subscribed systems increases
Last modified: 2013-10-01 06:52:13 EDT
Description of problem:
As the number of subscribed systems increases, the time to register/subscribe a new system increases dramatically. The increase from an unloaded system (0 systems) to one with 1000 systems, was 10 fold. The tests were run with a timeout of 120 seconds. When the system did time out, it appeared to be processing something in Candlepin (since the API call that was made is just proxied to Candlepin). The test was done using the same API calls that a subscription-manager client would when binding to a specify product id.
Version-Release number of selected component (if applicable):
Steps to Reproduce: (API calls)
1. GET /organizations/ACME_Corporation
2. POST /consumers/ (with facts)
3. PUT /consumers/:uuid/packages
4. POST /consumers/:uuid/entitlements (with product param)
Test system is 2 socket, 4 core, 2 threads per core, 24G RAM
Step 4 yielded 3 seconds on an unloaded system, 30+ sec on system with 1000 existing systems
What type of manifest did you have? 1 subscription with a quantity of 1000+, or lots of little?
That may be where it turns a little strange. The manifest I initially loaded only had a couple of subscriptions (3 at most). Now, there are 999 (or maybe more). While I think there is still some validity to this, I may need to run these tests again with a known good manifest.
If the subscriptions in the manifest are using virt_limit there would be one pool created for each system bind (which is for 4 guests on that specific host). This could explain the explosion in pools.
They should not however be affecting performance in recent versions of candlepin. (should be fixed since around candlepin-0.7.9-1)
The version of candlepin in this case is candlepin-0.7.8.1-1.el6cf.noarch.
I can re-run the tests with a later candlepin to see how it fares.
Upstream has been updated to candlepin-0.7.19. This contains the fix for this issue. When candlepin-0.7.19 is moved to a CFSE branch, this should be moved to ON_QA.
Moving all POST bugs to ON_QA since we have delivered a puddle with the bugs.
VERIFIED. Og Maciel registered and subscribed 1000 clients with no errors or no delay.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.