Bug 1734555

Description of problem:
Unable to Sync a package (RPM) which has lots of files. 

Version-Release number of selected component (if applicable):
Satellite 6.4 and Satellite 6.5

How reproducible:

100 %
Steps to Reproduce:
1. Create a rpm package with lots of file ( more than 100K ).
2. Create a custom product from Satellite web UI and create customer repository under that package. 
3. Upload the package using "hammer repository upload-content" command.

Actual results:
The sync will fail with an error similar to the following in /var/log/messages :

Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304) unexpected error occurred importing uploaded file: command document too large
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304) Traceback (most recent call last):
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/upload.py", line 120, in upload
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)     handlers[type_id](repo, type_id, unit_key, metadata, file_path, conduit, config)
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/upload.py", line 440, in _handle_package
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)     unit.save_and_import_content(file_path)
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)   File "/usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py", line 918, in save_and_import_content
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)     self.save()
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)   File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 324, in save
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)     object_id = collection.save(doc, **write_concern)
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 2180, in save
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)     check_keys, False, manipulate, write_concern)
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 709, in _update
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)     codec_options=self.codec_options).copy()
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)   File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 216, in command
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)     self._raise_connection_failure(error)
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)   File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 343, in _raise_connection_failure
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304)     raise error
Jul 31 02:32:35 satellite64 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: [8ce11872] (24085-90304) DocumentTooLarge: command document too large


Expected results:
No error and rpm should be synced without successfully.

Pulp 2 is in maintenance mode and currently accepts only critical/security issues. The main focus is on Pulp 3 and some of the requests will be satisfied in the newer version.
We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in Pulp 2. As this issue is not relevant for Pulp 3, we are therefore closing this out as WONTFIX.

The DocumentTooLarge issues have been seen in the past and there were multiple improvements in this area, all of which decreased the probability of hitting it.
To fully fix it in Pulp 2 would mean to redesign parts of the DB and potentially introduce data consistency problems because of the need to spread RPM metadata over multiple collections and the lack of proper transaction functionality in the db.
It was considered too risky even a while ago.

Please consider waiting for Pulp 3 which is not that far. It doesn't have mongodb and this problem is non-existent there.

Thank you for your interest in Satellite 6. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the product, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this feel free to contact your Red Hat Account Team. Thank you.