Bug 2188504
| Summary: | The "hammer export" command using single thread encryption causes a performance bottleneck. | |||
|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Brenden Wood <bwood> | |
| Component: | Pulp | Assignee: | satellite6-bugs <satellite6-bugs> | |
| Status: | CLOSED ERRATA | QA Contact: | Shweta Singh <shwsingh> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | high | |||
| Version: | Unspecified | CC: | ahumbe, dalley, dkliban, ggainey, hyu, iballou, jalviso, juqiao, osousa, pcreech, rchan, rlavi, shwsingh | |
| Target Milestone: | 6.14.0 | Keywords: | Performance, Triaged | |
| Target Release: | Unused | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | python-pulpcore-3.22.15-1.el8pc | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2238353 (view as bug list) | Environment: | ||
| Last Closed: | 2023-11-08 14:19:16 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
|
Description
Brenden Wood
2023-04-21 00:26:01 UTC
The default compression level of tarfile with gzip compression is level 9, the highest and most computationally intensive level. Possibly compression would be more viable at a lower level - based on various benchmarks level 3 ought to be about 4x faster than level 9 but with a compression level only 15-20% worse. Brendan, don't forget to attach the customer cases. If you have a reproducer (or a customer willing to experiment), what is the impact of adding "compresslevel=1" to that line? e.g. "with tarfile.open(tarfile_fp, "w|gz", compresslevel=1, fileobj=split_process.stdin)" (don't forget to restart the services, of course). Also, how large is the uncompressed export in question, in gigabytes? Hi Daniel, We have the ability to test this with a customer so I will try this out and report back. From memory, we were dealing with an export over a terabyte in size, but I will have to confirm that. Thanks Brendan, I thought I had posted this but apparently not - that patch actually will not work, because it requires code present in Python 3.12 only. Please don't ask the customer to try it just yet. The Pulp upstream bug status is at closed. Updating the external tracker on this bug. Anyone tracking this may also be interested in https://bugzilla.redhat.com/show_bug.cgi?id=2226950 Verified. Version Tested: Satellite 6.14.0 Snap 12.0 Verification Steps: 1. Enable some large repos like appstream and rhel7server. 2. Update the download policy to "immediate" and sync the repos. 3. Perform complete hammer export of the library lifecycle environment. 4. Observe the time to export the complete lce. Result: Performance of hammer export has been improved after the fix. Shweta, out of curiosity, how much improvement did you observe? I've unfortunately needed to revise the patch as it was not being reliably loaded 100% of the time. I don't believe there is any need to delay any scheduled releases, just push the BZ off to the next one. *** Bug 2238653 has been marked as a duplicate of this bug. *** *** Bug 2239183 has been marked as a duplicate of this bug. *** Hi Daniel! While exporting few large repos, I encountered "undefined method `first' for nil:NilClass" error. I have tried following steps: 1. Enable some large repos like appstream(Rhel8 and Rhel9) and rhel7server. 2. Update the download policy to "immediate" and sync the repos. 3. Perform complete hammer export of the library lifecycle environment. 4. Observe the time to export the complete lce. Version Tested: Satellite 6.14.0 Snap 17.0 Observation: During "hammer export" command, it failed with "undefined method `first' for nil:NilClass" error. That's a Ruby error, I'm not sure what would have caused it but it's unrelated to this BZ Is there any other information? Did the pulp task succeed or fail? If it failed that might have triggered some other Katello error which we're seeing, but if it succeeded or never ran in the first place then it is likely completely unrelated. Hi Daniel I was able to verify the fix and it was working. There is a significant improvement in the performance of the command("hammer export"). I have tested only on Python 3.11.
Verified.
Version Tested: Satellite 6.14.0 Snap 17
Verification Steps:
1. Enable few large repos(like appstream on rhel8 and rhel9 and rhel-7-server).
2. Update the download policy to immediate and sync all the repos.
3. Try complete export of the repos and notice the duration for the complete export.
Observation:
1. There is a significant improvement in the performance of the export.
2. It took ~23 min to complete export comapared to earlier taking 2 hours for the same export.
3. Repos which were exported: Appstream Rhel8, Appstream Rhel9 and Rhel-7-server.
4. All the code changes are present on the fixed version.
A hotfix is now available for Satellite 6.11.5 on RHEL 7 and RHEL 8. Please contact Red Hat support for installation instructions. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Satellite 6.14 security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:6818 |