Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1734555 - Sync failure - DocumentTooLarge: command document too large
Summary: Sync failure - DocumentTooLarge: command document too large
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.5.0
Hardware: Unspecified
OS: Linux
unspecified
medium
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Kersom
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-30 20:56 UTC by Gourav Padholia
Modified: 2024-10-01 16:19 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-09 17:02:24 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Comment 3 Gourav Padholia 2019-07-31 13:55:43 UTC
Description of problem:
Unable to Sync a package (RPM) which has lots of files. 

Version-Release number of selected component (if applicable):
Satellite 6.4 and Satellite 6.5

How reproducible:

100 %
Steps to Reproduce:
1. Create a rpm package with lots of file ( more than 100K ).
2. Create a custom product from Satellite web UI and create customer repository under that package. 
3. Upload the package using "hammer repository upload-content" command.

Actual results:
The sync will fail with an error similar to the following in /var/log/messages :

Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304) unexpected error occurred importing uploaded file: command document too large
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304) Traceback (most recent call last):
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/upload.py", line 120, in upload
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)     handlers[type_id](repo, type_id, unit_key, metadata, file_path, conduit, config)
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/upload.py", line 440, in _handle_package
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)     unit.save_and_import_content(file_path)
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)   File "/usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py", line 918, in save_and_import_content
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)     self.save()
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)   File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 324, in save
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)     object_id = collection.save(doc, **write_concern)
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 2180, in save
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)     check_keys, False, manipulate, write_concern)
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 709, in _update
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)     codec_options=self.codec_options).copy()
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)   File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 216, in command
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)     self._raise_connection_failure(error)
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)   File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 343, in _raise_connection_failure
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)     raise error
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304) DocumentTooLarge: command document too large


Expected results:
No error and rpm should be synced without successfully.

Comment 8 Tanya Tereshchenko 2020-05-01 13:55:31 UTC
Pulp 2 is in maintenance mode and currently accepts only critical/security issues. The main focus is on Pulp 3 and some of the requests will be satisfied in the newer version.
We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in Pulp 2. As this issue is not relevant for Pulp 3, we are therefore closing this out as WONTFIX.

Comment 11 Tanya Tereshchenko 2021-06-04 20:43:02 UTC
The DocumentTooLarge issues have been seen in the past and there were multiple improvements in this area, all of which decreased the probability of hitting it.
To fully fix it in Pulp 2 would mean to redesign parts of the DB and potentially introduce data consistency problems because of the need to spread RPM metadata over multiple collections and the lack of proper transaction functionality in the db.
It was considered too risky even a while ago.

Please consider waiting for Pulp 3 which is not that far. It doesn't have mongodb and this problem is non-existent there.

Comment 12 Mike McCune 2021-07-09 17:02:24 UTC
Thank you for your interest in Satellite 6. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the product, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this feel free to contact your Red Hat Account Team. Thank you.


Note You need to log in before you can comment on or make changes to this bug.