Bug 2074099

Summary: The errata migration continues to fail with "pymongo.errors.DocumentTooLarge: BSON document too large" error even after upgrading to Satellite 6.9.8
Product: Red Hat Satellite Reporter: Sayan Das <saydas>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Vladimír Sedmík <vsedmik>
Severity: high Docs Contact:
Priority: high    
Version: 6.9.8CC: ahumbe, dalley, dkliban, dleroux, ggainey, hyu, jpasqual, mjia, osousa, rchan, sadas, ttereshc, wclark
Target Milestone: 6.9.10Keywords: Triaged
Target Release: Unused   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: pulp-2to3-migration-0.11.12-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-17 17:17:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Hotfix RPM for Satellite 6.9.9 none

Description Sayan Das 2022-04-11 14:31:55 UTC 
Description of problem:

The following error was supposed to be fixed via Bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=2003888 but it still exists after upgrade to 6.9.8 when the count of ERRATA's are way to be migrated is very high.

~~~
pymongo.errors.DocumentTooLarge: BSON document too large (18938832 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.
~~~

This happens even if we configure the value of PULP_CONTENT_PREMIGRATION_BATCH_SIZE as low as 25 and repeatedly try.


Version-Release number of selected component (if applicable):

Satellite 6.9.8


How reproducible:

At times when ERRATA count is very high (assuming)


Steps to Reproduce:

Same as Bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=2003888.

Perhaps, sync a good amount of repo to have a huge amount of errata to be migrated during pulp 2 - pulp3 content migration and then attempt the same on a Satellite 6.9.8 


Actual results:

Mar 28 14:06:50 satellite69 pulpcore-worker-3: RuntimeWarning)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: pulp: rq.worker:ERROR: Traceback (most recent call last):
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Mar 28 14:06:50 satellite69 pulpcore-worker-3: rv = job.perform()
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Mar 28 14:06:50 satellite69 pulpcore-worker-3: self._result = self._execute()
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Mar 28 14:06:50 satellite69 pulpcore-worker-3: return self.func(*self.args, **self.kwargs)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 77, in migrate_from_pulp2
Mar 28 14:06:50 satellite69 pulpcore-worker-3: pre_migrate_all_content(plan)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 70, in pre_migrate_all_content
Mar 28 14:06:50 satellite69 pulpcore-worker-3: pre_migrate_content_type(content_model, mutable_type, lazy_type, premigrate_hook)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 301, in pre_migrate_content_type
Mar 28 14:06:50 satellite69 pulpcore-worker-3: record.id: record for record in batched_mongo_content_qs.no_cache()
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 300, in <dictcomp>
Mar 28 14:06:50 satellite69 pulpcore-worker-3: pulp2_content_by_id = {
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/mongoengine/queryset/base.py", line 1590, in __next__
Mar 28 14:06:50 satellite69 pulpcore-worker-3: raw_doc = next(self._cursor)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/cursor.py", line 1207, in next
Mar 28 14:06:50 satellite69 pulpcore-worker-3: if len(self.__data) or self._refresh():
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/cursor.py", line 1124, in _refresh
Mar 28 14:06:50 satellite69 pulpcore-worker-3: self.__send_message(q)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/cursor.py", line 1001, in __send_message
Mar 28 14:06:50 satellite69 pulpcore-worker-3: address=self.__address)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/mongo_client.py", line 1372, in _run_operation_with_response
Mar 28 14:06:50 satellite69 pulpcore-worker-3: exhaust=exhaust)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/mongo_client.py", line 1471, in _retryable_read
Mar 28 14:06:50 satellite69 pulpcore-worker-3: return func(session, server, sock_info, slave_ok)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/mongo_client.py", line 1366, in _cmd
Mar 28 14:06:50 satellite69 pulpcore-worker-3: unpack_res)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/server.py", line 116, in run_operation_with_response
Mar 28 14:06:50 satellite69 pulpcore-worker-3: sock_info.send_message(data, max_doc_size)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/pool.py", line 711, in send_message
Mar 28 14:06:50 satellite69 pulpcore-worker-3: (max_doc_size, self.max_bson_size))
Mar 28 14:06:50 satellite69 pulpcore-worker-3: pymongo.errors.DocumentTooLarge: BSON document too large (18938832 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.
Mar 28 14:06:50 satellite69 pulpcore-worker-3: Traceback (most recent call last):
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Mar 28 14:06:50 satellite69 pulpcore-worker-3: rv = job.perform()
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Mar 28 14:06:50 satellite69 pulpcore-worker-3: self._result = self._execute()
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Mar 28 14:06:50 satellite69 pulpcore-worker-3: return self.func(*self.args, **self.kwargs)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 77, in migrate_from_pulp2
Mar 28 14:06:50 satellite69 pulpcore-worker-3: pre_migrate_all_content(plan)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 70, in pre_migrate_all_content
Mar 28 14:06:50 satellite69 pulpcore-worker-3: pre_migrate_content_type(content_model, mutable_type, lazy_type, premigrate_hook)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 301, in pre_migrate_content_type
Mar 28 14:06:50 satellite69 pulpcore-worker-3: record.id: record for record in batched_mongo_content_qs.no_cache()
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 300, in <dictcomp>
Mar 28 14:06:50 satellite69 pulpcore-worker-3: pulp2_content_by_id = {
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/mongoengine/queryset/base.py", line 1590, in __next__
Mar 28 14:06:50 satellite69 pulpcore-worker-3: raw_doc = next(self._cursor)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/cursor.py", line 1207, in next
Mar 28 14:06:50 satellite69 pulpcore-worker-3: if len(self.__data) or self._refresh():
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/cursor.py", line 1124, in _refresh
Mar 28 14:06:50 satellite69 pulpcore-worker-3: self.__send_message(q)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/cursor.py", line 1001, in __send_message
Mar 28 14:06:50 satellite69 pulpcore-worker-3: address=self.__address)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/mongo_client.py", line 1372, in _run_operation_with_response
Mar 28 14:06:50 satellite69 pulpcore-worker-3: exhaust=exhaust)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/mongo_client.py", line 1471, in _retryable_read
Mar 28 14:06:50 satellite69 pulpcore-worker-3: return func(session, server, sock_info, slave_ok)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/mongo_client.py", line 1366, in _cmd
Mar 28 14:06:50 satellite69 pulpcore-worker-3: unpack_res)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/server.py", line 116, in run_operation_with_response
Mar 28 14:06:50 satellite69 pulpcore-worker-3: sock_info.send_message(data, max_doc_size)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/pool.py", line 711, in send_message
Mar 28 14:06:50 satellite69 pulpcore-worker-3: (max_doc_size, self.max_bson_size))
Mar 28 14:06:50 satellite69 pulpcore-worker-3: pymongo.errors.DocumentTooLarge: BSON document too large (18938832 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.


Expected results:

The satellite should be able properly to use the PULP_CONTENT_PREMIGRATION_BATCH_SIZE value and should be able to handle a large amount of ERRATA migration as well without any such errors.



Additional info:

Please check the private comment for some more details.

The only way I could help some of the customers here is by suggesting cleaning up unnecessary CV\CCV versions and repos that are not being used, followed by orphan cleanup i.e. by simply trying to reduce the amount of total errata to be migrated.
Comment 16 wclark 2022-08-29 20:09:17 UTC 
Created attachment 1908386 [details]
Hotfix RPM for Satellite 6.9.9

INSTALL INSTRUCTIONS:

1. Take a complete backup or snapshot of Satellite 6.9.9 server

2. Download the hotfix RPM attached to this BZ and copy it to Satellite server

3. # yum install ./python3-pulp-2to3-migration-0.11.10-2.HOTFIXRHBZ2074099RHBZ2038995.el7pc.noarch.rpm --disableplugin=foreman-protector
Comment 17 Hao Chang Yu 2022-09-07 07:51:22 UTC 
4. After installing the the hotfix rpm, restart the Satellite. foreman-maintain service restart
Comment 19 Vladimír Sedmík 2022-11-07 08:48:33 UTC 
Reproduced on 6.9.9 with python3-pulp-2to3-migration-0.11.10-1.el7pc.noarch
Verified on 6.9.10 snap 1 with python3-pulp-2to3-migration-0.11.12-1.el7pc.noarch

Steps to reproduce/verify:
1) Have a blank Sat with sufficient HW (24 CPUs, 64 GiB RAM, 500 GiB storage was enough) and manifest uploaded.
2) Sync one mid-size rpm repo (i.e. rhel-server-rhscl-7-rpms) and run initial content migration. It succeeds. 
3) Enable and sync 'many' large repos with 'big' erratas. I used these:
[root@sat ~]# hammer repository list --organization-id 1 --library 1 --fields Id,Name,Product
---|----------------------------------------------------------------------------------|--------------------------------------------------------------
ID | NAME                                                                             | PRODUCT                                                      
---|----------------------------------------------------------------------------------|--------------------------------------------------------------
23 | Red Hat Enterprise Linux 5 Server RPMs x86_64 5Server                            | Red Hat Enterprise Linux Server                              
22 | Red Hat Enterprise Linux 6 Server - Extended Life Cycle Support RPMs x86_64      | Red Hat Enterprise Linux Server - Extended Life Cycle Support
19 | Red Hat Enterprise Linux 7 Server Debug RPMs x86_64 7.8                          | Red Hat Enterprise Linux Server                              
20 | Red Hat Enterprise Linux 7 Server Debug RPMs x86_64 7.9                          | Red Hat Enterprise Linux Server                              
21 | Red Hat Enterprise Linux 7 Server Debug RPMs x86_64 7Server                      | Red Hat Enterprise Linux Server                              
6  | Red Hat Enterprise Linux 7 Server - Extended Update Support Debug RPMs x86_64... | Red Hat Enterprise Linux for x86_64 - Extended Update Support
7  | Red Hat Enterprise Linux 7 Server - Extended Update Support Debug RPMs x86_64... | Red Hat Enterprise Linux for x86_64 - Extended Update Support
8  | Red Hat Enterprise Linux 7 Server - Extended Update Support - Optional Debug ... | Red Hat Enterprise Linux for x86_64 - Extended Update Support
9  | Red Hat Enterprise Linux 7 Server - Extended Update Support - Optional Debug ... | Red Hat Enterprise Linux for x86_64 - Extended Update Support
4  | Red Hat Enterprise Linux 7 Server - Extended Update Support - Optional RPMs x... | Red Hat Enterprise Linux for x86_64 - Extended Update Support
5  | Red Hat Enterprise Linux 7 Server - Extended Update Support - Optional RPMs x... | Red Hat Enterprise Linux for x86_64 - Extended Update Support
2  | Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.6      | Red Hat Enterprise Linux for x86_64 - Extended Update Support
3  | Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7      | Red Hat Enterprise Linux for x86_64 - Extended Update Support
16 | Red Hat Enterprise Linux 7 Server - Optional Debug RPMs x86_64 7.8               | Red Hat Enterprise Linux Server                              
17 | Red Hat Enterprise Linux 7 Server - Optional Debug RPMs x86_64 7.9               | Red Hat Enterprise Linux Server                              
18 | Red Hat Enterprise Linux 7 Server - Optional Debug RPMs x86_64 7Server           | Red Hat Enterprise Linux Server                              
10 | Red Hat Enterprise Linux 7 Server - Optional RPMs x86_64 7.8                     | Red Hat Enterprise Linux Server                              
11 | Red Hat Enterprise Linux 7 Server - Optional RPMs x86_64 7.9                     | Red Hat Enterprise Linux Server                              
12 | Red Hat Enterprise Linux 7 Server - Optional RPMs x86_64 7Server                 | Red Hat Enterprise Linux Server                              
13 | Red Hat Enterprise Linux 7 Server RPMs x86_64 7.8                                | Red Hat Enterprise Linux Server                              
14 | Red Hat Enterprise Linux 7 Server RPMs x86_64 7.9                                | Red Hat Enterprise Linux Server                              
15 | Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server                            | Red Hat Enterprise Linux Server                              
1  | Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 Server x86_6... | Red Hat Software Collections (for RHEL Server)               
---|----------------------------------------------------------------------------------|--------------------------------------------------------------
4) Create a Content View and add all the repos into it.
5) Copy the CV into 19 clones, so now we have 20 CVs full of big repos.
6) Publish all the CVs and promote them to one more LCE. This way we get around 4.5M erratas ready.
7) Start the content migration once again and check /var/log/messages for errors (this will take some time and logfile might get rotated).

On 6.9.9 the issue was replicated and migration failed:
...
Nov  4 16:12:02 sat pulpcore-worker-4: File "/usr/lib64/python3.6/site-packages/pymongo/server.py", line 116, in run_operation_with_response
Nov  4 16:12:02 sat pulpcore-worker-4: sock_info.send_message(data, max_doc_size)
Nov  4 16:12:02 sat pulpcore-worker-4: File "/usr/lib64/python3.6/site-packages/pymongo/pool.py", line 711, in send_message
Nov  4 16:12:02 sat pulpcore-worker-4: (max_doc_size, self.max_bson_size))
Nov  4 16:12:02 sat pulpcore-worker-4: pymongo.errors.DocumentTooLarge: BSON document too large (76560653 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.

On 6.9.10 snap the migration succeeded, no errors spotted in messages:
[root@sat ~]# satellite-maintain content migration-stats
Running Retrieve Pulp 2 to Pulp 3 migration statistics
================================================================================
Retrieve Pulp 2 to Pulp 3 migration statistics: 
============Migration Summary================
Migrated/Total RPMs: 144370/144370
Migrated/Total errata: 4514244/4514244
Migrated/Total repositories: 1403/1403
Estimated migration time based on yum content, fast hardware, and low server load: 1 hours, 38 minutes
Estimated migration time based on yum content, slow hardware, and high server load: 6 hours, 32 minutes

Note: ensure there is sufficient storage space for /var/lib/pulp/published to double in size before starting the migration process.
Check the size of /var/lib/pulp/published with 'du -sh /var/lib/pulp/published/'

Note: ensure there is sufficient storage space for postgresql.
You will need additional space for your postgresql database.  The partition holding '/var/opt/rh/rh-postgresql12/lib/pgsql/data/'
   will need additional free space equivalent to the size of your Mongo db database (/var/lib/mongodb/).
                                                                      [OK]
Comment 25 errata-xmlrpc 2022-11-17 17:17:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.9.10 Async Security Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8532
Comment 26 Red Hat Bugzilla 2023-09-18 04:35:15 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days