Bug 1919418

Summary: [candlepin] Unable to import subscription manifest in large environment (60K+ consumers/hosts)
Product: Red Hat Satellite Reporter: James Jeffers <jjeffers>
Component: CandlepinAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Danny Synk <dsynk>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.7.0CC: egolov, ehelms, hhudgeon, jjeffers, nmoumoul, pcreech, zhunting
Target Milestone: 6.8.4Keywords: PrioBumpField, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: candlepin-3.1.24 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1840805 Environment:
Last Closed: 2021-02-24 12:47:13 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Comment 2 Danny Synk 2021-02-16 19:16:12 UTC
Steps to retest:

1. Configure a Satellite with 60K entitled Candlepin consumers.

2. Double the default values for Tomcat's Java heap memory minimum and maximum:

    # grep JAVA_OPTS /etc/tomcat/tomcat.conf
    JAVA_OPTS="-Xms2048m -Xmx8192m"

3. Refresh the manifest:

    # time hammer subscription refresh-manifest --organization-id 1
    [.................................................] [100%]

    real    39m6.116s                                                                                                                                             │
    user    0m9.633s                                                                                                                                              │
    sys     0m1.498s

4. Delete the manifest:

    # time hammer subscription delete-manifest --organization-id 1
    [.................................................] [100%]

    real    0m10.218s
    user    0m1.659s
    sys     0m0.251s

5. Import a manifest:

    # time hammer subscription upload --file manifest_1919417_20210211T160313Z.zip --organization-id 1
    warning: Overriding "Content-Type" header "multipart/form-data" with "multipart/form-data; boundary=----RubyFormBoundaryOvdKUh5RbWUDpBGk" due to payload
    [.............................................................................................................] [100%]

    real    0m9.427s
    user    0m1.737s
    sys     0m0.206s

Expected Results:
On a Satellite with 60K or more Candlepin consumers, refreshing, deleting, and importing a manifest should all complete successfully in a timely fashion.

Actual Results:
Manifest refresh completes in under 40 minutes (vs. encountering "Error: Runtime Error This connection has been closed. at org.postgresql.jdbc.PgConnection.checkClosed:767" on Satellite 6.8.2). Manifest delete and import complete in approximately ten seconds each.

Verified on Satellite 6.8.4, snap 1 (candlepin 3.1.24-1).

Comment 7 errata-xmlrpc 2021-02-24 12:47:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite 6.8.4 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0654