Bug 2003888 - The Pulp 2 to Pulp 3 migration stucks and raises 'pymongo.errors.DocumentTooLarge' error in /var/log/messages
Summary: The Pulp 2 to Pulp 3 migration stucks and raises 'pymongo.errors.DocumentTooL...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.9.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: 6.9.8
Assignee: satellite6-bugs
QA Contact: Lai
URL:
Whiteboard:
: 2032684 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-09-14 05:38 UTC by Hao Chang Yu
Modified: 2022-09-07 16:00 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-01-27 17:33:09 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Pulp Redmine 9482 0 Normal CLOSED - CURRENTRELEASE Backport "Set bach_size to mitigate query result exceeding 16mb" to 0.11.z 2021-11-17 23:08:37 UTC
Red Hat Knowledge Base (Solution) 6620261 0 None None None 2022-01-04 07:31:11 UTC
Red Hat Product Errata RHBA-2022:0320 0 None None None 2022-01-27 17:33:17 UTC

Description Hao Chang Yu 2021-09-14 05:38:03 UTC
Description of problem:
The Pulp 2 to Pulp 3 migration stucks and raises 'pymongo.errors.DocumentTooLarge' error in /var/log/messages. The strange thing is, it only happened once. The error is gone after re-running the 'satellite-maintain content prepare'. I also tried migration-reset and run migration again but unable to reproduce this issue the 2nd time. I think this might happen to some users having large number of errata so I created this bugzilla for further investigation if needed.

# satellite-maintain content prepare
...
Prepare content for Pulp 3: 
Checking for valid Katello configuraton.
Starting task.
2021-08-31 15:01:37 +1000: Pre-migrating Pulp 2 erratum content (detail info) 601855/602875


pulpcore-worker-4: RuntimeWarning)
pulpcore-worker-4: pulp: rq.worker:ERROR: Traceback (most recent call last):
pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
pulpcore-worker-4: rv = job.perform()
pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
pulpcore-worker-4: self._result = self._execute()
pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
pulpcore-worker-4: return self.func(*self.args, **self.kwargs)
pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 77, in migrate_from_pulp2
pulpcore-worker-4: pre_migrate_all_content(plan)
pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 70, in pre_migrate_all_content
pulpcore-worker-4: pre_migrate_content_type(content_model, mutable_type, lazy_type, premigrate_hook)
pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 300, in pre_migrate_content_type
pulpcore-worker-4: record.id: record for record in mongo_content_qs.only(*mongo_fields).no_cache()
pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 299, in <dictcomp>
pulpcore-worker-4: pulp2_content_by_id = {
pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/mongoengine/queryset/base.py", line 1590, in __next__
pulpcore-worker-4: raw_doc = next(self._cursor)
pulpcore-worker-4: File "/usr/lib64/python3.6/site-packages/pymongo/cursor.py", line 1207, in next
pulpcore-worker-4: if len(self.__data) or self._refresh():
pulpcore-worker-4: File "/usr/lib64/python3.6/site-packages/pymongo/cursor.py", line 1124, in _refresh
pulpcore-worker-4: self.__send_message(q)
pulpcore-worker-4: File "/usr/lib64/python3.6/site-packages/pymongo/cursor.py", line 1001, in __send_message
pulpcore-worker-4: address=self.__address)
pulpcore-worker-4: File "/usr/lib64/python3.6/site-packages/pymongo/mongo_client.py", line 1372, in _run_operation_with_response
pulpcore-worker-4: exhaust=exhaust)
pulpcore-worker-4: File "/usr/lib64/python3.6/site-packages/pymongo/mongo_client.py", line 1471, in _retryable_read
pulpcore-worker-4: return func(session, server, sock_info, slave_ok)
pulpcore-worker-4: File "/usr/lib64/python3.6/site-packages/pymongo/mongo_client.py", line 1366, in _cmd
pulpcore-worker-4: unpack_res)
pulpcore-worker-4: File "/usr/lib64/python3.6/site-packages/pymongo/server.py", line 116, in run_operation_with_response
pulpcore-worker-4: sock_info.send_message(data, max_doc_size)
pulpcore-worker-4: File "/usr/lib64/python3.6/site-packages/pymongo/pool.py", line 711, in send_message
pulpcore-worker-4: (max_doc_size, self.max_bson_size))
pulpcore-worker-4: pymongo.errors.DocumentTooLarge: BSON document too large (32444638 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.

Comment 1 Ina Panova 2021-09-23 14:45:31 UTC
Without steps to reproduce or more information about the repos and the content that was migrating I am not sure how we can proceed.

Comment 4 pulp-infra@redhat.com 2021-09-27 12:12:38 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 5 pulp-infra@redhat.com 2021-09-27 12:12:39 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 6 pulp-infra@redhat.com 2021-10-02 18:08:35 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 7 pulp-infra@redhat.com 2021-10-02 19:06:34 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 8 pulp-infra@redhat.com 2021-10-02 19:06:35 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 9 pulp-infra@redhat.com 2021-10-02 20:08:01 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 10 pulp-infra@redhat.com 2021-10-02 21:06:45 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 11 pulp-infra@redhat.com 2021-11-17 23:08:38 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 12 Brad Buckingham 2021-12-16 16:08:54 UTC
*** Bug 2032684 has been marked as a duplicate of this bug. ***

Comment 13 Lai 2022-01-06 14:30:11 UTC
Note: Since this issue is difficult to reproduce, my testing is limited.  I was only able to get around 8.5k erratas in my testing.

Steps to retest
1. Sync several repos with a lot of erratas
2. Run the 'satellite-maintain content prepare'
3. Ensure that migration is successful

Expected result:
Migration should be successful

Actual result:
Migration is successful

Verified on 6.9.8_01

Comment 19 errata-xmlrpc 2022-01-27 17:33:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite 6.9.8 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0320


Note You need to log in before you can comment on or make changes to this bug.