Bug 869286 - SCALE: Subscription of systems gets slower and slower as number of subscribed systems increases
SCALE: Subscription of systems gets slower and slower as number of subscribed...
Product: Subscription Asset Manager
Classification: Red Hat
Component: candlepin (Show other bugs)
Unspecified Unspecified
unspecified Severity medium
: rc
: 1.3
Assigned To: candlepin-bugs
Katello QA List
: Triaged, ZStream
Depends On:
Blocks: sam13-tracker 896261
  Show dependency treegraph
Reported: 2012-10-23 09:38 EDT by Brad P. Crochet
Modified: 2013-10-01 06:52 EDT (History)
9 users (show)

See Also:
Fixed In Version: candlepin-0.7.19
Doc Type: Release Note
Doc Text:
A performance issue in the candlepin component causes subscription responsiveness to decrease as the number of systems subscribed to CloudForms System Engine increases. Red Hat Engineering aims to resolve this issue in a future update to Red Hat CloudForms.
Story Points: ---
Clone Of:
: 896261 (view as bug list)
Last Closed: 2013-10-01 06:52:13 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Brad P. Crochet 2012-10-23 09:38:28 EDT
Description of problem:
As the number of subscribed systems increases, the time to register/subscribe a new system increases dramatically. The increase from an unloaded system (0 systems) to one with 1000 systems, was 10 fold. The tests were run with a timeout of 120 seconds. When the system did time out, it appeared to be processing something in Candlepin (since the API call that was made is just proxied to Candlepin). The test was done using the same API calls that a subscription-manager client would when binding to a specify product id.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce: (API calls)
1. GET /organizations/ACME_Corporation
2. POST /consumers/ (with facts)
3. PUT /consumers/:uuid/packages
4. POST /consumers/:uuid/entitlements (with product param)
Actual results:
Test system is 2 socket, 4 core, 2 threads per core, 24G RAM
Step 4 yielded 3 seconds on an unloaded system, 30+ sec on system with 1000 existing systems
Comment 2 Bryan Kearney 2012-10-23 10:33:29 EDT
What type of manifest did you have? 1 subscription with a quantity of 1000+, or lots of little?
Comment 3 Brad P. Crochet 2012-10-23 13:52:39 EDT
That may be where it turns a little strange. The manifest I initially loaded only had a couple of subscriptions (3 at most). Now, there are 999 (or maybe more). While I think there is still some validity to this, I may need to run these tests again with a known good manifest.
Comment 4 Devan Goodwin 2012-10-23 14:13:54 EDT
If the subscriptions in the manifest are using virt_limit there would be one pool created for each system bind (which is for 4 guests on that specific host). This could explain the explosion in pools.

They should not however be affecting performance in recent versions of candlepin. (should be fixed since around candlepin-0.7.9-1)
Comment 5 Brad P. Crochet 2012-10-24 08:15:48 EDT
The version of candlepin in this case is candlepin-

I can re-run the tests with a later candlepin to see how it fares.
Comment 7 Bryan Kearney 2012-11-29 10:08:53 EST
Upstream has been updated to candlepin-0.7.19. This contains the fix for this issue. When candlepin-0.7.19 is moved to a CFSE branch, this should be moved to ON_QA.
Comment 9 Bryan Kearney 2013-06-07 14:20:22 EDT
Moving all POST bugs to ON_QA since we have delivered a puddle with the bugs.
Comment 11 sthirugn@redhat.com 2013-08-15 12:23:06 EDT
VERIFIED. Og Maciel registered and subscribed 1000 clients with no errors or no delay.

Packages tested:
* candlepin-0.8.19-1.el6sam.noarch
* candlepin-scl-1-5.el6_4.noarch
* candlepin-scl-quartz-2.1.5-5.el6_4.noarch
* candlepin-scl-rhino-1.7R3-1.el6_4.noarch
* candlepin-scl-runtime-1-5.el6_4.noarch
* candlepin-selinux-0.8.19-1.el6sam.noarch
* candlepin-tomcat6-0.8.19-1.el6sam.noarch
* elasticsearch-0.19.9-8.el6sat.noarch
* katello-candlepin-cert-key-pair-1.0-1.noarch
* katello-certs-tools-1.4.2-2.el6sat.noarch
* katello-cli-1.4.3-5.el6sat.noarch
* katello-cli-common-1.4.3-5.el6sat.noarch
* katello-common-1.4.3-6.el6sam_splice.noarch
* katello-configure-1.4.4-2.el6sat.noarch
* katello-glue-candlepin-1.4.3-6.el6sam_splice.noarch
* katello-glue-elasticsearch-1.4.3-6.el6sam_splice.noarch
* katello-headpin-1.4.3-6.el6sam_splice.noarch
* katello-headpin-all-1.4.3-6.el6sam_splice.noarch
* katello-selinux-1.4.4-2.el6sat.noarch
* thumbslug-0.0.32-1.el6sam.noarch
* thumbslug-selinux-0.0.32-1.el6sam.noarch
Comment 13 errata-xmlrpc 2013-10-01 06:52:13 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.