Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Witnessed same behavior when performing a complete export on 6.10.1.0. Openned a support case (03099228) on Dec 8. How did you get Pulp to print the Exception type?
Created attachment 1852518[details]
python3-pulpcore-3.14.9-2.HOTFIXRHBZ2033847.el7pc.noarch.rpm
HOTFIX RPM is available for Satellite 6.10.2
INSTALL INSTRUCTIONS:
1. Take a complete backup or snapshot of Satellite 6.10.1 server
2. Download the hotfix RPM attached to this BZ and copy it to Satellite server
3. # yum install ./python3-pulpcore-3.14.9-2.HOTFIXRHBZ2033847.el7pc.noarch.rpm --disableplugin=foreman-protector
4. # systemctl restart pulpcore-{content,api,worker@*}.service
Comment 19Gary Scarborough
2022-02-03 00:33:47 UTC
*** Bug 2048876 has been marked as a duplicate of this bug. ***
Steps to test:
Prereq: Ensure you have enough disk space.
1. Enable and sync:
Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server
2. Create a content view and add the above repo
3. Publish the content view
4. Do a full export the content view
5. Publish a new version of the cv
6. Do an incremental export
Expected:
Complete and incremental export should finish successfully
Actual:
Complete and incremental export does finish successfully
# hammer content-export complete version --content-view="export-cv" --organization-id=1 --lifecycle-environment Library
[................................................................................................................................] [100%]
[root@dhcp-3-9 ~]# hammer content-export incremental version --content-view="export-cv" --version=2.0 --organization-id=1 --lifecycle-environment Library --chunk-size-gb 2
[................................................................................................................................] [100%]
Verified on 7.0 snap 10 with python38-pulpcore-3.16.3-1.el8pc.noarch. This is tested on both rhel7.9 and rhel8.5
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2022:5498
Description of problem: Pulp raises memory error when exporting repositories with large contents. hammer content-export incremental version --content-view=rhel --version=2.0 --organization="Default Organization" --lifecycle-environment Library --chunk-size-gb 2 [....................................................................................................................................................................] [100%] Error: Pulp task error undefined method `first' for nil:NilClas # Traceback in /var/log/messages ---------------------------------------------- pulpcore-worker-7: pulp [029c96b2-78a8-48ca-8605-9d59872f1be0]: pulpcore.tasking.pulpcore_worker:INFO: Task 508915bc-d4f5-49e3-aedb-d838bb0b2d14 failed () <========= Raised error without message which is normally MemoryError ... pulpcore-worker-7: pulp [029c96b2-78a8-48ca-8605-9d59872f1be0]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task pulpcore-worker-7: result = func(*args, **kwargs) pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 204, in pulp_export pulpcore-worker-7: _do_export(pulp_exporter, tar, the_export) pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 313, in _do_export pulpcore-worker-7: export_content(the_export, version) pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 134, in export_content pulpcore-worker-7: _write_export(export.tarfile, resource, dest_dir) pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 41, in _write_export pulpcore-worker-7: data = dataset.json.encode("utf8") pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/tablib/formats/__init__.py", line 62, in __get__ pulpcore-worker-7: return self._format.export_set(obj, **kwargs) pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/tablib/formats/_json.py", line 27, in export_set pulpcore-worker-7: dataset.dict, default=serialize_objects_handler, ensure_ascii=False pulpcore-worker-7: File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps pulpcore-worker-7: **kw).encode(obj) pulpcore-worker-7: File "/usr/lib64/python3.6/json/encoder.py", line 202, in encode pulpcore-worker-7: return ''.join(chunks) ---------------------------------------------- # Add a logging to Pulp so that it will print the Exception type; ---------------------------------------------- pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO: Task b246ce3c-a805-4a0f-8393-2fbf7cdd73a7 failed () pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task pulpcore-worker-6: result = func(*args, **kwargs) pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 204, in pulp_export pulpcore-worker-6: _do_export(pulp_exporter, tar, the_export) pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 313, in _do_export pulpcore-worker-6: export_content(the_export, version) pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 134, in export_content pulpcore-worker-6: _write_export(export.tarfile, resource, dest_dir) pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 41, in _write_export pulpcore-worker-6: data = dataset.json.encode("utf8") pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO: <class 'MemoryError'> <======= ---------------------------------------------- # While the task is running, we can see that the memory consumption is very high total used free shared buff/cache available Mem: 33010716 19796992 221744 579536 12991980 12230452 Swap: 12058620 3917188 8141432 total used free shared buff/cache available Mem: 33010716 20831688 331752 579536 11847276 11196116 Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 21926716 224460 579540 10859540 10100732 Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 23173152 219596 579540 9617968 8854304 Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 24317448 238176 579660 8455092 7709900 Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 25737804 230592 579612 7042320 6289292 Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 27344320 226584 579612 5439812 4683004 Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 30193200 234504 579612 2583012 1834164 <=============== Observed free memory dropped to 1.8GB from 20GB Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 24422544 7876732 334776 711440 7886420 Swap: 12058620 8729032 3329588 <===================== Free swap dropped 3G total used free shared buff/cache available Mem: 33010716 23213200 9030436 336976 767080 9067412 Swap: 12058620 8674504 3384116 total used free shared buff/cache available Mem: 33010716 22871236 9357400 339484 782080 9400532 Swap: 12058620 8637896 3420724 total used free shared buff/cache available Mem: 33010716 7216620 25034040 339872 760056 25053004 <======== Memory freed Swap: 12058620 8431560 3627060 total used free shared buff/cache available Mem: 33010716 7234448 25014092 339928 762176 25033868 Swap: 12058620 8414408 3644212 # As we can see that Pulp worker consumed 22.3G of RAM when writing the resource file top - 00:31:24 up 1 day, 12:52, 6 users, load average: 60.50, 26.92, 13.02 Tasks: 373 total, 1 running, 372 sleeping, 0 stopped, 0 zombie %Cpu0 : 1.3 us, 9.3 sy, 0.0 ni, 0.0 id, 89.3 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu1 : 0.3 us, 1.7 sy, 0.0 ni, 0.0 id, 97.7 wa, 0.0 hi, 0.0 si, 0.3 st %Cpu2 : 0.3 us, 2.3 sy, 0.0 ni, 0.0 id, 97.0 wa, 0.0 hi, 0.3 si, 0.0 st %Cpu3 : 1.7 us, 5.0 sy, 0.0 ni, 0.0 id, 93.4 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu4 : 0.3 us, 1.0 sy, 0.0 ni, 0.0 id, 98.7 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu5 : 0.7 us, 3.0 sy, 0.0 ni, 0.0 id, 96.3 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu6 : 1.0 us, 4.3 sy, 0.0 ni, 0.0 id, 94.6 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu7 : 3.0 us, 7.7 sy, 0.0 ni, 0.0 id, 89.3 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 99.8/33010716 [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||] KiB Swap: 52.1/12058620 [|||||||||||||||||||||||||||||||||||||||||||||||||||| ] PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8442 pulp 20 0 23.9g 22.3g 636 D 17.2 70.7 13:38.63 pulpcore-worker <============= # And it consumes 10G of RAM when writing a smaller file (I think) Tasks: 369 total, 3 running, 366 sleeping, 0 stopped, 0 zombie %Cpu0 : 1.7 us, 1.0 sy, 0.0 ni, 75.2 id, 22.2 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu1 : 33.2 us, 2.3 sy, 0.0 ni, 62.8 id, 1.3 wa, 0.0 hi, 0.3 si, 0.0 st %Cpu2 : 17.5 us, 2.0 sy, 0.0 ni, 62.0 id, 18.5 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu3 : 26.1 us, 6.4 sy, 0.0 ni, 34.4 id, 33.1 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu4 : 45.8 us, 5.6 sy, 0.0 ni, 46.2 id, 2.0 wa, 0.0 hi, 0.0 si, 0.3 st %Cpu5 : 4.4 us, 3.7 sy, 0.0 ni, 68.2 id, 23.6 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu6 : 1.3 us, 0.7 sy, 0.0 ni, 91.7 id, 6.3 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu7 : 70.6 us, 1.7 sy, 0.0 ni, 27.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 56.3/33010716 [|||||||||||||||||||||||||||||||||||||||||||||||||||||||| ] KiB Swap: 34.2/12058620 [|||||||||||||||||||||||||||||||||| ] PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8442 pulp 20 0 10.4g 10.0g 856 R 99.7 31.7 17:25.61 pulpcore-worker <============= Steps to Reproduce: 1. Enable and sync the following 2 repos Red_Hat_Enterprise_Linux_6_Server_-_Extended_Life_Cycle_Support_RPMs_x86_64 Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server 2. Create a content view and add the above repos 3. Publish the content view 4. Export the content view hammer content-export incremental version --content-view=rhel --version=1.0 --organization="Default Organization" --lifecycle-environment Library --chunk-size-gb 2 Actual results: Error: Pulp task error undefined method `first' for nil:NilClass Expected results: Success