Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1287129 - Manifest import gets really slow for increasing number of entitlements in one pool
Summary: Manifest import gets really slow for increasing number of entitlements in one...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Candlepin
Version: 6.1.3
Hardware: x86_64
OS: Linux
high
high
Target Milestone: Unspecified
Assignee: Barnaby Court
QA Contact: Lukas Pramuk
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-12-01 14:51 UTC by Pavel Moravec
Modified: 2020-07-16 08:39 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-27 09:21:11 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1501 0 normal SHIPPED_LIVE Red Hat Satellite 6.2 Capsule and Server 2016-07-27 12:28:58 UTC

Description Pavel Moravec 2015-12-01 14:51:35 UTC 
Description of problem:
Increasing number of entitlements in a single pool in manifest causes import time of the manifest to grow very rapidly. I.e. for manifest with 15k subscriptions, it took several minutes, but for manifest with 30k subscriptions, it didnt finish within 45 minutes(sic!).

Checking candlepin logs, the cause is:

2015-12-01 07:40:44,026 [req=6868c8d9-121c-496b-a603-41c4c8847ae5, org=customer] INFO  org.candlepin.controller.CandlepinPoolManager - Revoked entitlement: 2c9281204f5ff656014ffcf7fbc46d38

takes more and more time each. Having just partial logs (all to be attached), it ended after 45 minutes of importing in revoking one entitlement in 2 seconds(!).

Since the time of processing one entitlement grows substantially over time, it is assumed some Map/Hash-like structure used in candlepin is very inefficient on insert or update when growing over some limit.

Note also a valid workaround is to split the entitlements into more pools.


Version-Release number of selected component (if applicable):
seen on 6.0.8, supposed to be elsewhere


How reproducible:
100% (expected, just seen at customer)


Steps to Reproduce:
1. Create a manifest with 2x15k entitlements (15k entitlements in 2 pools each).
2. Import the manifest to Satellite
3. Create a manifest with 30k entitlements in single pool
4. Try to import it to the Satellite (after increasing rest_client_timeout in /etc/foreman/plugins/katello.yaml)


Actual results:
2. succeeds, 4. doesnt wthin 45 minutes


Expected results:
both to succeed within similar time


Additional info:
Comment 5 Barnaby Court 2016-02-26 19:36:38 UTC 
This should be fixed in Sat 6.1.7
Comment 7 Lukas Pramuk 2016-04-26 11:12:44 UTC 
VERIFIED.

@Sat6.2.0-Beta-Snap9.2 

# time hammer subscription upload --organization "Testing Organization" --file newest_problematic_manifest.zip
[..............................................................................................................] [100%]


>>> real	0m15.797s
...

# hammer subscription list --organization "Testing Organization"
-----|----------|---------|---------|----------|----------|----------|----|---------|----------|---------
NAME | CONTRACT | ACCOUNT | SUPPORT | QUANTITY | CONSUMED | END DATE | ID | PRODUCT | QUANTITY | ATTACHED
-----|----------|---------|---------|----------|----------|----------|----|---------|----------|---------

>>> none subs imported yet, lets refresh manifest

# time hammer subscription refresh-manifest --organization "Testing Organization"
[..............................................................................................................] [100%]


>>> real	0m23.887s
...

# hammer subscription list --organization "Testing Organization"--------------------------------------------------------|----------|---------|---------|----------|----------|------------------------------|----|---------------------------------------------------------|----------|---------
NAME                                                    | CONTRACT | ACCOUNT  | SUPPORT | QUANTITY | CONSUMED | END DATE                     | ID | PRODUCT                                                 | QUANTITY | ATTACHED
--------------------------------------------------------|----------|----------|---------|----------|----------|------------------------------|----|---------------------------------------------------------|----------|---------
Red Hat Infrastructure Foundation Layer - for Providers | <edited> | <edited> | Premium | 100      | 0        | 2017-03-01T04:59:59.000+0000 | 34 | Red Hat Infrastructure Foundation Layer - for Providers | 100      | 0       
Red Hat Infrastructure Foundation Layer - for Providers | <edited> | <edited> | Premium | 12000    | 0        | 2017-03-01T04:59:59.000+0000 | 32 | Red Hat Infrastructure Foundation Layer - for Providers | 12000    | 0       
Red Hat Infrastructure Foundation Layer - for Providers | <edited> | <edited> | Premium | 5000     | 0        | 2017-03-01T04:59:59.000+0000 | 30 | Red Hat Infrastructure Foundation Layer - for Providers | 5000     | 0       
Red Hat Infrastructure Foundation Layer - for Providers | <edited> | <edited> | Premium | 10000    | 0        | 2017-03-01T04:59:59.000+0000 | 28 | Red Hat Infrastructure Foundation Layer - for Providers | 10000    | 0       
Red Hat Infrastructure Foundation Layer - for Providers | <edited> | <edited> | Premium | 3800     | 0        | 2017-03-01T04:59:59.000+0000 | 26 | Red Hat Infrastructure Foundation Layer - for Providers | 3800     | 0       
--------------------------------------------------------|----------|---------|---------|----------|----------|------------------------------|----|---------------------------------------------------------|----------|---------

>>> subs (of problematic manifest) are imported successfully within a minute 
>>> however for big subs count it seems it's a two step process now: after import (16sec) it needs a refresh (24sec) - lets have another bz for that
Comment 9 errata-xmlrpc 2016-07-27 09:21:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501
Note You need to log in before you can comment on or make changes to this bug.