2074099 – The errata migration continues to fail with "pymongo.errors.DocumentTooLarge: BSON document too large" error even after upgrading to Satellite 6.9.8

Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2074099 - The errata migration continues to fail with "pymongo.errors.DocumentTooLarge: BSON document too large" error even after upgrading to Satellite 6.9.8

Summary: The errata migration continues to fail with "pymongo.errors.DocumentTooLarge:...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Satellite
Classification:	Red Hat
Component:	Pulp
Sub Component:
Version:	6.9.8
Hardware:	All
OS:	All
Priority:	high
Severity:	high
Target Milestone:	6.9.10
Assignee:	satellite6-bugs
QA Contact:	Vladimír Sedmík
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-04-11 14:31 UTC by Sayan Das
Modified:	2023-09-18 04:35 UTC (History)
CC List:	13 users (show)
Fixed In Version:	pulp-2to3-migration-0.11.12-1
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-11-17 17:17:17 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
Hotfix RPM for Satellite 6.9.9 (225.80 KB, application/x-rpm) 2022-08-29 20:09 UTC, wclark	no flags	Details
View All

Links
System	ID	Priority	Status	Summary	Last Updated
Github	pulp pulp-2to3-migration pull 576	None	Merged	Fix BSON too large error when pre-migrating many errata	2022-08-17 15:32:21 UTC
Red Hat Knowledge Base (Solution)	6620261	None	None	None	2022-04-11 14:34:35 UTC
Red Hat Product Errata	RHSA-2022:8532	None	None	None	2022-11-17 17:17:30 UTC

Description Sayan Das 2022-04-11 14:31:55 UTC

Description of problem:

The following error was supposed to be fixed via Bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=2003888 but it still exists after upgrade to 6.9.8 when the count of ERRATA's are way to be migrated is very high.

~~~
pymongo.errors.DocumentTooLarge: BSON document too large (18938832 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.
~~~

This happens even if we configure the value of PULP_CONTENT_PREMIGRATION_BATCH_SIZE as low as 25 and repeatedly try.


Version-Release number of selected component (if applicable):

Satellite 6.9.8


How reproducible:

At times when ERRATA count is very high (assuming)


Steps to Reproduce:

Same as Bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=2003888.

Perhaps, sync a good amount of repo to have a huge amount of errata to be migrated during pulp 2 - pulp3 content migration and then attempt the same on a Satellite 6.9.8 


Actual results:

Mar 28 14:06:50 satellite69 pulpcore-worker-3: RuntimeWarning)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: pulp: rq.worker:ERROR: Traceback (most recent call last):
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Mar 28 14:06:50 satellite69 pulpcore-worker-3: rv = job.perform()
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Mar 28 14:06:50 satellite69 pulpcore-worker-3: self._result = self._execute()
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Mar 28 14:06:50 satellite69 pulpcore-worker-3: return self.func(*self.args, **self.kwargs)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 77, in migrate_from_pulp2
Mar 28 14:06:50 satellite69 pulpcore-worker-3: pre_migrate_all_content(plan)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 70, in pre_migrate_all_content
Mar 28 14:06:50 satellite69 pulpcore-worker-3: pre_migrate_content_type(content_model, mutable_type, lazy_type, premigrate_hook)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 301, in pre_migrate_content_type
Mar 28 14:06:50 satellite69 pulpcore-worker-3: record.id: record for record in batched_mongo_content_qs.no_cache()
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 300, in <dictcomp>
Mar 28 14:06:50 satellite69 pulpcore-worker-3: pulp2_content_by_id = {
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/mongoengine/queryset/base.py", line 1590, in __next__
Mar 28 14:06:50 satellite69 pulpcore-worker-3: raw_doc = next(self._cursor)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/cursor.py", line 1207, in next
Mar 28 14:06:50 satellite69 pulpcore-worker-3: if len(self.__data) or self._refresh():
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/cursor.py", line 1124, in _refresh
Mar 28 14:06:50 satellite69 pulpcore-worker-3: self.__send_message(q)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/cursor.py", line 1001, in __send_message
Mar 28 14:06:50 satellite69 pulpcore-worker-3: address=self.__address)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/mongo_client.py", line 1372, in _run_operation_with_response
Mar 28 14:06:50 satellite69 pulpcore-worker-3: exhaust=exhaust)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/mongo_client.py", line 1471, in _retryable_read
Mar 28 14:06:50 satellite69 pulpcore-worker-3: return func(session, server, sock_info, slave_ok)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/mongo_client.py", line 1366, in _cmd
Mar 28 14:06:50 satellite69 pulpcore-worker-3: unpack_res)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/server.py", line 116, in run_operation_with_response
Mar 28 14:06:50 satellite69 pulpcore-worker-3: sock_info.send_message(data, max_doc_size)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/pool.py", line 711, in send_message
Mar 28 14:06:50 satellite69 pulpcore-worker-3: (max_doc_size, self.max_bson_size))
Mar 28 14:06:50 satellite69 pulpcore-worker-3: pymongo.errors.DocumentTooLarge: BSON document too large (18938832 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.
Mar 28 14:06:50 satellite69 pulpcore-worker-3: Traceback (most recent call last):
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Mar 28 14:06:50 satellite69 pulpcore-worker-3: rv = job.perform()
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Mar 28 14:06:50 satellite69 pulpcore-worker-3: self._result = self._execute()
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Mar 28 14:06:50 satellite69 pulpcore-worker-3: return self.func(*self.args, **self.kwargs)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 77, in migrate_from_pulp2
Mar 28 14:06:50 satellite69 pulpcore-worker-3: pre_migrate_all_content(plan)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 70, in pre_migrate_all_content
Mar 28 14:06:50 satellite69 pulpcore-worker-3: pre_migrate_content_type(content_model, mutable_type, lazy_type, premigrate_hook)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 301, in pre_migrate_content_type
Mar 28 14:06:50 satellite69 pulpcore-worker-3: record.id: record for record in batched_mongo_content_qs.no_cache()
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 300, in <dictcomp>
Mar 28 14:06:50 satellite69 pulpcore-worker-3: pulp2_content_by_id = {
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/mongoengine/queryset/base.py", line 1590, in __next__
Mar 28 14:06:50 satellite69 pulpcore-worker-3: raw_doc = next(self._cursor)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/cursor.py", line 1207, in next
Mar 28 14:06:50 satellite69 pulpcore-worker-3: if len(self.__data) or self._refresh():
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/cursor.py", line 1124, in _refresh
Mar 28 14:06:50 satellite69 pulpcore-worker-3: self.__send_message(q)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/cursor.py", line 1001, in __send_message
Mar 28 14:06:50 satellite69 pulpcore-worker-3: address=self.__address)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/mongo_client.py", line 1372, in _run_operation_with_response
Mar 28 14:06:50 satellite69 pulpcore-worker-3: exhaust=exhaust)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/mongo_client.py", line 1471, in _retryable_read
Mar 28 14:06:50 satellite69 pulpcore-worker-3: return func(session, server, sock_info, slave_ok)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/mongo_client.py", line 1366, in _cmd
Mar 28 14:06:50 satellite69 pulpcore-worker-3: unpack_res)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/server.py", line 116, in run_operation_with_response
Mar 28 14:06:50 satellite69 pulpcore-worker-3: sock_info.send_message(data, max_doc_size)
Mar 28 14:06:50 satellite69 pulpcore-worker-3: File "/usr/lib64/python3.6/site-packages/pymongo/pool.py", line 711, in send_message
Mar 28 14:06:50 satellite69 pulpcore-worker-3: (max_doc_size, self.max_bson_size))
Mar 28 14:06:50 satellite69 pulpcore-worker-3: pymongo.errors.DocumentTooLarge: BSON document too large (18938832 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.


Expected results:

The satellite should be able properly to use the PULP_CONTENT_PREMIGRATION_BATCH_SIZE value and should be able to handle a large amount of ERRATA migration as well without any such errors.



Additional info:

Please check the private comment for some more details.

The only way I could help some of the customers here is by suggesting cleaning up unnecessary CV\CCV versions and repos that are not being used, followed by orphan cleanup i.e. by simply trying to reduce the amount of total errata to be migrated.

Comment 16 wclark 2022-08-29 20:09:17 UTC

Created attachment 1908386 [details]
Hotfix RPM for Satellite 6.9.9

INSTALL INSTRUCTIONS:

1. Take a complete backup or snapshot of Satellite 6.9.9 server

2. Download the hotfix RPM attached to this BZ and copy it to Satellite server

3. # yum install ./python3-pulp-2to3-migration-0.11.10-2.HOTFIXRHBZ2074099RHBZ2038995.el7pc.noarch.rpm --disableplugin=foreman-protector

Comment 17 Hao Chang Yu 2022-09-07 07:51:22 UTC

4. After installing the the hotfix rpm, restart the Satellite. foreman-maintain service restart

Comment 19 Vladimír Sedmík 2022-11-07 08:48:33 UTC

Reproduced on 6.9.9 with python3-pulp-2to3-migration-0.11.10-1.el7pc.noarch
Verified on 6.9.10 snap 1 with python3-pulp-2to3-migration-0.11.12-1.el7pc.noarch

Steps to reproduce/verify:
1) Have a blank Sat with sufficient HW (24 CPUs, 64 GiB RAM, 500 GiB storage was enough) and manifest uploaded.
2) Sync one mid-size rpm repo (i.e. rhel-server-rhscl-7-rpms) and run initial content migration. It succeeds. 
3) Enable and sync 'many' large repos with 'big' erratas. I used these:
[root@sat ~]# hammer repository list --organization-id 1 --library 1 --fields Id,Name,Product
---|----------------------------------------------------------------------------------|--------------------------------------------------------------
ID | NAME                                                                             | PRODUCT                                                      
---|----------------------------------------------------------------------------------|--------------------------------------------------------------
23 | Red Hat Enterprise Linux 5 Server RPMs x86_64 5Server                            | Red Hat Enterprise Linux Server                              
22 | Red Hat Enterprise Linux 6 Server - Extended Life Cycle Support RPMs x86_64      | Red Hat Enterprise Linux Server - Extended Life Cycle Support
19 | Red Hat Enterprise Linux 7 Server Debug RPMs x86_64 7.8                          | Red Hat Enterprise Linux Server                              
20 | Red Hat Enterprise Linux 7 Server Debug RPMs x86_64 7.9                          | Red Hat Enterprise Linux Server                              
21 | Red Hat Enterprise Linux 7 Server Debug RPMs x86_64 7Server                      | Red Hat Enterprise Linux Server                              
6  | Red Hat Enterprise Linux 7 Server - Extended Update Support Debug RPMs x86_64... | Red Hat Enterprise Linux for x86_64 - Extended Update Support
7  | Red Hat Enterprise Linux 7 Server - Extended Update Support Debug RPMs x86_64... | Red Hat Enterprise Linux for x86_64 - Extended Update Support
8  | Red Hat Enterprise Linux 7 Server - Extended Update Support - Optional Debug ... | Red Hat Enterprise Linux for x86_64 - Extended Update Support
9  | Red Hat Enterprise Linux 7 Server - Extended Update Support - Optional Debug ... | Red Hat Enterprise Linux for x86_64 - Extended Update Support
4  | Red Hat Enterprise Linux 7 Server - Extended Update Support - Optional RPMs x... | Red Hat Enterprise Linux for x86_64 - Extended Update Support
5  | Red Hat Enterprise Linux 7 Server - Extended Update Support - Optional RPMs x... | Red Hat Enterprise Linux for x86_64 - Extended Update Support
2  | Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.6      | Red Hat Enterprise Linux for x86_64 - Extended Update Support
3  | Red Hat Enterprise Linux 7 Server - Extended Update Support RPMs x86_64 7.7      | Red Hat Enterprise Linux for x86_64 - Extended Update Support
16 | Red Hat Enterprise Linux 7 Server - Optional Debug RPMs x86_64 7.8               | Red Hat Enterprise Linux Server                              
17 | Red Hat Enterprise Linux 7 Server - Optional Debug RPMs x86_64 7.9               | Red Hat Enterprise Linux Server                              
18 | Red Hat Enterprise Linux 7 Server - Optional Debug RPMs x86_64 7Server           | Red Hat Enterprise Linux Server                              
10 | Red Hat Enterprise Linux 7 Server - Optional RPMs x86_64 7.8                     | Red Hat Enterprise Linux Server                              
11 | Red Hat Enterprise Linux 7 Server - Optional RPMs x86_64 7.9                     | Red Hat Enterprise Linux Server                              
12 | Red Hat Enterprise Linux 7 Server - Optional RPMs x86_64 7Server                 | Red Hat Enterprise Linux Server                              
13 | Red Hat Enterprise Linux 7 Server RPMs x86_64 7.8                                | Red Hat Enterprise Linux Server                              
14 | Red Hat Enterprise Linux 7 Server RPMs x86_64 7.9                                | Red Hat Enterprise Linux Server                              
15 | Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server                            | Red Hat Enterprise Linux Server                              
1  | Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 Server x86_6... | Red Hat Software Collections (for RHEL Server)               
---|----------------------------------------------------------------------------------|--------------------------------------------------------------
4) Create a Content View and add all the repos into it.
5) Copy the CV into 19 clones, so now we have 20 CVs full of big repos.
6) Publish all the CVs and promote them to one more LCE. This way we get around 4.5M erratas ready.
7) Start the content migration once again and check /var/log/messages for errors (this will take some time and logfile might get rotated).

On 6.9.9 the issue was replicated and migration failed:
...
Nov  4 16:12:02 sat pulpcore-worker-4: File "/usr/lib64/python3.6/site-packages/pymongo/server.py", line 116, in run_operation_with_response
Nov  4 16:12:02 sat pulpcore-worker-4: sock_info.send_message(data, max_doc_size)
Nov  4 16:12:02 sat pulpcore-worker-4: File "/usr/lib64/python3.6/site-packages/pymongo/pool.py", line 711, in send_message
Nov  4 16:12:02 sat pulpcore-worker-4: (max_doc_size, self.max_bson_size))
Nov  4 16:12:02 sat pulpcore-worker-4: pymongo.errors.DocumentTooLarge: BSON document too large (76560653 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.

On 6.9.10 snap the migration succeeded, no errors spotted in messages:
[root@sat ~]# satellite-maintain content migration-stats
Running Retrieve Pulp 2 to Pulp 3 migration statistics
================================================================================
Retrieve Pulp 2 to Pulp 3 migration statistics: 
============Migration Summary================
Migrated/Total RPMs: 144370/144370
Migrated/Total errata: 4514244/4514244
Migrated/Total repositories: 1403/1403
Estimated migration time based on yum content, fast hardware, and low server load: 1 hours, 38 minutes
Estimated migration time based on yum content, slow hardware, and high server load: 6 hours, 32 minutes

Note: ensure there is sufficient storage space for /var/lib/pulp/published to double in size before starting the migration process.
Check the size of /var/lib/pulp/published with 'du -sh /var/lib/pulp/published/'

Note: ensure there is sufficient storage space for postgresql.
You will need additional space for your postgresql database.  The partition holding '/var/opt/rh/rh-postgresql12/lib/pgsql/data/'
   will need additional free space equivalent to the size of your Mongo db database (/var/lib/mongodb/).
                                                                      [OK]

Comment 25 errata-xmlrpc 2022-11-17 17:17:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.9.10 Async Security Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8532

Comment 26 Red Hat Bugzilla 2023-09-18 04:35:15 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days

Note You need to log in before you can comment on or make changes to this bug.