Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2033847 - Content view export failed with undefined method `first' for nil:NilClass
Summary: Content view export failed with undefined method `first' for nil:NilClass
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.10.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: 6.11.0
Assignee: Hao Chang Yu
QA Contact: Lai
URL:
Whiteboard:
: 2033731 2048876 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-12-18 04:23 UTC by Hao Chang Yu
Modified: 2022-07-29 09:07 UTC (History)
24 users (show)

Fixed In Version: tfm-pulpcore-python-pulpcore-3.16.3
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2043699 (view as bug list)
Environment:
Last Closed: 2022-07-05 14:31:10 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
python3-pulpcore-3.14.9-2.HOTFIXRHBZ2033847.el7pc.noarch.rpm (623.59 KB, application/x-rpm)
2022-01-21 14:45 UTC, wclark 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Github pulp pulpcore pull 1782 0 None Merged Fix import and export OOM error 2022-01-19 12:49:06 UTC
Pulp Redmine 9645 0 None None None 2021-12-19 11:24:02 UTC
Red Hat Knowledge Base (Solution) 6676651 0 None None None 2022-01-27 05:55:00 UTC
Red Hat Product Errata RHSA-2022:5498 0 None None None 2022-07-05 14:31:47 UTC

Description Hao Chang Yu 2021-12-18 04:23:44 UTC 
Description of problem:
Pulp raises memory error when exporting repositories with large contents.

hammer content-export incremental version --content-view=rhel --version=2.0 --organization="Default Organization" --lifecycle-environment Library --chunk-size-gb 2
[....................................................................................................................................................................] [100%]
Error: Pulp task error
undefined method `first' for nil:NilClas


# Traceback in /var/log/messages
----------------------------------------------
pulpcore-worker-7: pulp [029c96b2-78a8-48ca-8605-9d59872f1be0]: pulpcore.tasking.pulpcore_worker:INFO: Task 508915bc-d4f5-49e3-aedb-d838bb0b2d14 failed ()  <========= Raised error without message which is normally MemoryError
...
pulpcore-worker-7: pulp [029c96b2-78a8-48ca-8605-9d59872f1be0]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
pulpcore-worker-7: result = func(*args, **kwargs)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 204, in pulp_export
pulpcore-worker-7: _do_export(pulp_exporter, tar, the_export)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 313, in _do_export
pulpcore-worker-7: export_content(the_export, version)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 134, in export_content
pulpcore-worker-7: _write_export(export.tarfile, resource, dest_dir)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 41, in _write_export
pulpcore-worker-7: data = dataset.json.encode("utf8")
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/tablib/formats/__init__.py", line 62, in __get__
pulpcore-worker-7: return self._format.export_set(obj, **kwargs)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/tablib/formats/_json.py", line 27, in export_set
pulpcore-worker-7: dataset.dict, default=serialize_objects_handler, ensure_ascii=False
pulpcore-worker-7: File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
pulpcore-worker-7: **kw).encode(obj)
pulpcore-worker-7: File "/usr/lib64/python3.6/json/encoder.py", line 202, in encode
pulpcore-worker-7: return ''.join(chunks)
----------------------------------------------


# Add a logging to Pulp so that it will print the Exception type;
----------------------------------------------
pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO: Task b246ce3c-a805-4a0f-8393-2fbf7cdd73a7 failed ()
pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
pulpcore-worker-6: result = func(*args, **kwargs)
pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 204, in pulp_export
pulpcore-worker-6: _do_export(pulp_exporter, tar, the_export)
pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 313, in _do_export
pulpcore-worker-6: export_content(the_export, version)
pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 134, in export_content
pulpcore-worker-6: _write_export(export.tarfile, resource, dest_dir)
pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 41, in _write_export
pulpcore-worker-6: data = dataset.json.encode("utf8")
pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO: <class 'MemoryError'>  <=======
----------------------------------------------


# While the task is running, we can see that the memory consumption is very high

              total        used        free      shared  buff/cache   available
Mem:       33010716    19796992      221744      579536    12991980    12230452
Swap:      12058620     3917188     8141432

              total        used        free      shared  buff/cache   available
Mem:       33010716    20831688      331752      579536    11847276    11196116
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    21926716      224460      579540    10859540    10100732
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    23173152      219596      579540     9617968     8854304
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    24317448      238176      579660     8455092     7709900
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    25737804      230592      579612     7042320     6289292
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    27344320      226584      579612     5439812     4683004
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    30193200      234504      579612     2583012     1834164    <=============== Observed free memory dropped to 1.8GB from 20GB
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    24422544     7876732      334776      711440     7886420
Swap:      12058620     8729032     3329588             <===================== Free swap dropped 3G

              total        used        free      shared  buff/cache   available
Mem:       33010716    23213200     9030436      336976      767080     9067412
Swap:      12058620     8674504     3384116

              total        used        free      shared  buff/cache   available
Mem:       33010716    22871236     9357400      339484      782080     9400532
Swap:      12058620     8637896     3420724

              total        used        free      shared  buff/cache   available
Mem:       33010716     7216620    25034040      339872      760056    25053004   <======== Memory freed
Swap:      12058620     8431560     3627060

              total        used        free      shared  buff/cache   available
Mem:       33010716     7234448    25014092      339928      762176    25033868
Swap:      12058620     8414408     3644212



# As we can see that Pulp worker consumed 22.3G of RAM when writing the resource file

top - 00:31:24 up 1 day, 12:52,  6 users,  load average: 60.50, 26.92, 13.02
Tasks: 373 total,   1 running, 372 sleeping,   0 stopped,   0 zombie
%Cpu0  :  1.3 us,  9.3 sy,  0.0 ni,  0.0 id, 89.3 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.3 us,  1.7 sy,  0.0 ni,  0.0 id, 97.7 wa,  0.0 hi,  0.0 si,  0.3 st
%Cpu2  :  0.3 us,  2.3 sy,  0.0 ni,  0.0 id, 97.0 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu3  :  1.7 us,  5.0 sy,  0.0 ni,  0.0 id, 93.4 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  :  0.3 us,  1.0 sy,  0.0 ni,  0.0 id, 98.7 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu5  :  0.7 us,  3.0 sy,  0.0 ni,  0.0 id, 96.3 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu6  :  1.0 us,  4.3 sy,  0.0 ni,  0.0 id, 94.6 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu7  :  3.0 us,  7.7 sy,  0.0 ni,  0.0 id, 89.3 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 99.8/33010716 [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||]
KiB Swap: 52.1/12058620 [||||||||||||||||||||||||||||||||||||||||||||||||||||                                                ]

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                            
 8442 pulp      20   0   23.9g  22.3g    636 D  17.2 70.7  13:38.63 pulpcore-worker  <=============


# And it consumes 10G of RAM when writing a smaller file (I think)

Tasks: 369 total,   3 running, 366 sleeping,   0 stopped,   0 zombie
%Cpu0  :  1.7 us,  1.0 sy,  0.0 ni, 75.2 id, 22.2 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  : 33.2 us,  2.3 sy,  0.0 ni, 62.8 id,  1.3 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu2  : 17.5 us,  2.0 sy,  0.0 ni, 62.0 id, 18.5 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  : 26.1 us,  6.4 sy,  0.0 ni, 34.4 id, 33.1 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  : 45.8 us,  5.6 sy,  0.0 ni, 46.2 id,  2.0 wa,  0.0 hi,  0.0 si,  0.3 st
%Cpu5  :  4.4 us,  3.7 sy,  0.0 ni, 68.2 id, 23.6 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu6  :  1.3 us,  0.7 sy,  0.0 ni, 91.7 id,  6.3 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu7  : 70.6 us,  1.7 sy,  0.0 ni, 27.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 56.3/33010716 [||||||||||||||||||||||||||||||||||||||||||||||||||||||||                                            ]
KiB Swap: 34.2/12058620 [||||||||||||||||||||||||||||||||||                                                                  ]

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                  
 8442 pulp      20   0   10.4g  10.0g    856 R  99.7 31.7  17:25.61 pulpcore-worker  <=============


Steps to Reproduce:
1. Enable and sync the following 2 repos

Red_Hat_Enterprise_Linux_6_Server_-_Extended_Life_Cycle_Support_RPMs_x86_64
Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server

2. Create a content view and add the above repos
3. Publish the content view
4. Export the content view

hammer content-export incremental version --content-view=rhel --version=1.0 --organization="Default Organization" --lifecycle-environment Library --chunk-size-gb 2


Actual results:
Error: Pulp task error
undefined method `first' for nil:NilClass

Expected results:
Success
Comment 3 William Skupa 2021-12-21 22:11:50 UTC 
Witnessed same behavior when performing a complete export on 6.10.1.0.  Openned a support case (03099228) on Dec 8.  How did you get Pulp to print the Exception type?
Comment 18 wclark 2022-01-21 14:45:07 UTC 
Created attachment 1852518 [details]
python3-pulpcore-3.14.9-2.HOTFIXRHBZ2033847.el7pc.noarch.rpm

HOTFIX RPM is available for Satellite 6.10.2

INSTALL INSTRUCTIONS:

1. Take a complete backup or snapshot of Satellite 6.10.1 server

2. Download the hotfix RPM attached to this BZ and copy it to Satellite server

3. # yum install ./python3-pulpcore-3.14.9-2.HOTFIXRHBZ2033847.el7pc.noarch.rpm --disableplugin=foreman-protector

4. # systemctl restart pulpcore-{content,api,worker@*}.service
Comment 19 Gary Scarborough 2022-02-03 00:33:47 UTC 
*** Bug 2048876 has been marked as a duplicate of this bug. ***
Comment 21 Lai 2022-02-22 16:51:42 UTC 
Steps to test:

Prereq: Ensure you have enough disk space.

1. Enable and sync:

Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server

2. Create a content view and add the above repo
3. Publish the content view
4. Do a full export the content view
5. Publish a new version of the cv
6. Do an incremental export

Expected:
Complete and incremental export should finish successfully

Actual:
Complete and incremental export does finish successfully

# hammer content-export complete version --content-view="export-cv" --organization-id=1 --lifecycle-environment Library
[................................................................................................................................] [100%]

[root@dhcp-3-9 ~]# hammer content-export incremental version --content-view="export-cv" --version=2.0 --organization-id=1 --lifecycle-environment Library --chunk-size-gb 2
[................................................................................................................................] [100%]

Verified on 7.0 snap 10 with python38-pulpcore-3.16.3-1.el8pc.noarch.  This is tested on both rhel7.9 and rhel8.5
Comment 27 Partha Aji 2022-06-29 17:44:29 UTC 
*** Bug 2033731 has been marked as a duplicate of this bug. ***
Comment 29 errata-xmlrpc 2022-07-05 14:31:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5498
Note You need to log in before you can comment on or make changes to this bug.