Bug 2033847
| Summary: | Content view export failed with undefined method `first' for nil:NilClass | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Hao Chang Yu <hyu> | ||||
| Component: | Pulp | Assignee: | Hao Chang Yu <hyu> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Lai <ltran> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 6.10.0 | CC: | ahumbe, dkliban, ggainey, gscarbor, jalviso, jbhatia, jpasqual, jrichards2, kkinge, mhaverkamp, mshimura, msunil, paji, pcreech, pratshar, rchan, rcoe, rmynar, sadas, saydas, tasander, wclark, william.skupaiii, yoliynyk | ||||
| Target Milestone: | 6.11.0 | Keywords: | PrioBumpGSS, Triaged | ||||
| Target Release: | Unused | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | tfm-pulpcore-python-pulpcore-3.16.3 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | |||||||
| : | 2043699 (view as bug list) | Environment: | |||||
| Last Closed: | 2022-07-05 14:31:10 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Witnessed same behavior when performing a complete export on 6.10.1.0. Openned a support case (03099228) on Dec 8. How did you get Pulp to print the Exception type? Created attachment 1852518 [details]
python3-pulpcore-3.14.9-2.HOTFIXRHBZ2033847.el7pc.noarch.rpm
HOTFIX RPM is available for Satellite 6.10.2
INSTALL INSTRUCTIONS:
1. Take a complete backup or snapshot of Satellite 6.10.1 server
2. Download the hotfix RPM attached to this BZ and copy it to Satellite server
3. # yum install ./python3-pulpcore-3.14.9-2.HOTFIXRHBZ2033847.el7pc.noarch.rpm --disableplugin=foreman-protector
4. # systemctl restart pulpcore-{content,api,worker@*}.service
*** Bug 2048876 has been marked as a duplicate of this bug. *** Steps to test: Prereq: Ensure you have enough disk space. 1. Enable and sync: Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server 2. Create a content view and add the above repo 3. Publish the content view 4. Do a full export the content view 5. Publish a new version of the cv 6. Do an incremental export Expected: Complete and incremental export should finish successfully Actual: Complete and incremental export does finish successfully # hammer content-export complete version --content-view="export-cv" --organization-id=1 --lifecycle-environment Library [................................................................................................................................] [100%] [root@dhcp-3-9 ~]# hammer content-export incremental version --content-view="export-cv" --version=2.0 --organization-id=1 --lifecycle-environment Library --chunk-size-gb 2 [................................................................................................................................] [100%] Verified on 7.0 snap 10 with python38-pulpcore-3.16.3-1.el8pc.noarch. This is tested on both rhel7.9 and rhel8.5 *** Bug 2033731 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5498 |
Description of problem: Pulp raises memory error when exporting repositories with large contents. hammer content-export incremental version --content-view=rhel --version=2.0 --organization="Default Organization" --lifecycle-environment Library --chunk-size-gb 2 [....................................................................................................................................................................] [100%] Error: Pulp task error undefined method `first' for nil:NilClas # Traceback in /var/log/messages ---------------------------------------------- pulpcore-worker-7: pulp [029c96b2-78a8-48ca-8605-9d59872f1be0]: pulpcore.tasking.pulpcore_worker:INFO: Task 508915bc-d4f5-49e3-aedb-d838bb0b2d14 failed () <========= Raised error without message which is normally MemoryError ... pulpcore-worker-7: pulp [029c96b2-78a8-48ca-8605-9d59872f1be0]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task pulpcore-worker-7: result = func(*args, **kwargs) pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 204, in pulp_export pulpcore-worker-7: _do_export(pulp_exporter, tar, the_export) pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 313, in _do_export pulpcore-worker-7: export_content(the_export, version) pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 134, in export_content pulpcore-worker-7: _write_export(export.tarfile, resource, dest_dir) pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 41, in _write_export pulpcore-worker-7: data = dataset.json.encode("utf8") pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/tablib/formats/__init__.py", line 62, in __get__ pulpcore-worker-7: return self._format.export_set(obj, **kwargs) pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/tablib/formats/_json.py", line 27, in export_set pulpcore-worker-7: dataset.dict, default=serialize_objects_handler, ensure_ascii=False pulpcore-worker-7: File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps pulpcore-worker-7: **kw).encode(obj) pulpcore-worker-7: File "/usr/lib64/python3.6/json/encoder.py", line 202, in encode pulpcore-worker-7: return ''.join(chunks) ---------------------------------------------- # Add a logging to Pulp so that it will print the Exception type; ---------------------------------------------- pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO: Task b246ce3c-a805-4a0f-8393-2fbf7cdd73a7 failed () pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task pulpcore-worker-6: result = func(*args, **kwargs) pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 204, in pulp_export pulpcore-worker-6: _do_export(pulp_exporter, tar, the_export) pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 313, in _do_export pulpcore-worker-6: export_content(the_export, version) pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 134, in export_content pulpcore-worker-6: _write_export(export.tarfile, resource, dest_dir) pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 41, in _write_export pulpcore-worker-6: data = dataset.json.encode("utf8") pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO: <class 'MemoryError'> <======= ---------------------------------------------- # While the task is running, we can see that the memory consumption is very high total used free shared buff/cache available Mem: 33010716 19796992 221744 579536 12991980 12230452 Swap: 12058620 3917188 8141432 total used free shared buff/cache available Mem: 33010716 20831688 331752 579536 11847276 11196116 Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 21926716 224460 579540 10859540 10100732 Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 23173152 219596 579540 9617968 8854304 Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 24317448 238176 579660 8455092 7709900 Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 25737804 230592 579612 7042320 6289292 Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 27344320 226584 579612 5439812 4683004 Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 30193200 234504 579612 2583012 1834164 <=============== Observed free memory dropped to 1.8GB from 20GB Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 24422544 7876732 334776 711440 7886420 Swap: 12058620 8729032 3329588 <===================== Free swap dropped 3G total used free shared buff/cache available Mem: 33010716 23213200 9030436 336976 767080 9067412 Swap: 12058620 8674504 3384116 total used free shared buff/cache available Mem: 33010716 22871236 9357400 339484 782080 9400532 Swap: 12058620 8637896 3420724 total used free shared buff/cache available Mem: 33010716 7216620 25034040 339872 760056 25053004 <======== Memory freed Swap: 12058620 8431560 3627060 total used free shared buff/cache available Mem: 33010716 7234448 25014092 339928 762176 25033868 Swap: 12058620 8414408 3644212 # As we can see that Pulp worker consumed 22.3G of RAM when writing the resource file top - 00:31:24 up 1 day, 12:52, 6 users, load average: 60.50, 26.92, 13.02 Tasks: 373 total, 1 running, 372 sleeping, 0 stopped, 0 zombie %Cpu0 : 1.3 us, 9.3 sy, 0.0 ni, 0.0 id, 89.3 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu1 : 0.3 us, 1.7 sy, 0.0 ni, 0.0 id, 97.7 wa, 0.0 hi, 0.0 si, 0.3 st %Cpu2 : 0.3 us, 2.3 sy, 0.0 ni, 0.0 id, 97.0 wa, 0.0 hi, 0.3 si, 0.0 st %Cpu3 : 1.7 us, 5.0 sy, 0.0 ni, 0.0 id, 93.4 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu4 : 0.3 us, 1.0 sy, 0.0 ni, 0.0 id, 98.7 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu5 : 0.7 us, 3.0 sy, 0.0 ni, 0.0 id, 96.3 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu6 : 1.0 us, 4.3 sy, 0.0 ni, 0.0 id, 94.6 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu7 : 3.0 us, 7.7 sy, 0.0 ni, 0.0 id, 89.3 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 99.8/33010716 [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||] KiB Swap: 52.1/12058620 [|||||||||||||||||||||||||||||||||||||||||||||||||||| ] PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8442 pulp 20 0 23.9g 22.3g 636 D 17.2 70.7 13:38.63 pulpcore-worker <============= # And it consumes 10G of RAM when writing a smaller file (I think) Tasks: 369 total, 3 running, 366 sleeping, 0 stopped, 0 zombie %Cpu0 : 1.7 us, 1.0 sy, 0.0 ni, 75.2 id, 22.2 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu1 : 33.2 us, 2.3 sy, 0.0 ni, 62.8 id, 1.3 wa, 0.0 hi, 0.3 si, 0.0 st %Cpu2 : 17.5 us, 2.0 sy, 0.0 ni, 62.0 id, 18.5 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu3 : 26.1 us, 6.4 sy, 0.0 ni, 34.4 id, 33.1 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu4 : 45.8 us, 5.6 sy, 0.0 ni, 46.2 id, 2.0 wa, 0.0 hi, 0.0 si, 0.3 st %Cpu5 : 4.4 us, 3.7 sy, 0.0 ni, 68.2 id, 23.6 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu6 : 1.3 us, 0.7 sy, 0.0 ni, 91.7 id, 6.3 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu7 : 70.6 us, 1.7 sy, 0.0 ni, 27.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 56.3/33010716 [|||||||||||||||||||||||||||||||||||||||||||||||||||||||| ] KiB Swap: 34.2/12058620 [|||||||||||||||||||||||||||||||||| ] PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8442 pulp 20 0 10.4g 10.0g 856 R 99.7 31.7 17:25.61 pulpcore-worker <============= Steps to Reproduce: 1. Enable and sync the following 2 repos Red_Hat_Enterprise_Linux_6_Server_-_Extended_Life_Cycle_Support_RPMs_x86_64 Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server 2. Create a content view and add the above repos 3. Publish the content view 4. Export the content view hammer content-export incremental version --content-view=rhel --version=1.0 --organization="Default Organization" --lifecycle-environment Library --chunk-size-gb 2 Actual results: Error: Pulp task error undefined method `first' for nil:NilClass Expected results: Success