Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1404345 - Sync failure - DocumentTooLarge
Summary: Sync failure - DocumentTooLarge
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.2.4
Hardware: Unspecified
OS: Unspecified
urgent
urgent with 1 vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: jcallaha
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-12-13 16:03 UTC by Brad Buckingham
Modified: 2021-06-10 11:44 UTC (History)
22 users (show)

Fixed In Version: pulp-rpm-2.8.7.9-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-01 13:56:59 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Patch to compress RPM metadata, compatible with pulp_rpm 2.8.7 (26.51 KB, patch)
2017-02-27 17:52 UTC, Tanya Tereshchenko 		no flags Details | Diff
Updated patch to compress RPM metadata for pulp_rpm 2.8.7 (27.79 KB, patch)
2017-02-27 20:28 UTC, Tanya Tereshchenko 		no flags Details | Diff
Patch to compress RPM metadata with re-numbered migration for pulp_rpm 2.8.7 (26.69 KB, patch)
2017-03-13 22:48 UTC, Tanya Tereshchenko 		no flags Details | Diff
Show Obsolete (2) View All


Links
System ID Private Priority Status Summary Last Updated
Pulp Redmine 723 0 High CLOSED - CURRENTRELEASE RPMs with large number of files can exceed mongo document size limit 2017-04-19 21:37:21 UTC
Pulp Redmine 2652 0 Normal CLOSED - COMPLETE Backport #723, RPMs with large number of files can exceed mongo document size limit 2017-03-30 23:06:52 UTC
Red Hat Knowledge Base (Solution) 2957371 0 None None None 2017-03-07 15:10:06 UTC
Red Hat Product Errata RHBA-2017:1191 0 normal SHIPPED_LIVE Satellite 6.2.9 Async Bug Release 2017-05-01 17:49:42 UTC

Description Brad Buckingham 2016-12-13 16:03:02 UTC 
Description of problem:

Unable to sync a repository that has RPMs that are 'too big' (e.g. 1+ GB).

Version-Release number of selected component (if applicable):

Satellite 6.2

How reproducible:

Always, if the RPM is over the limit

Steps to Reproduce:

1. create a repo (e.g. https://fedorapeople.org/groups/katello/releases/yum/foreman/1.13/el6/x86_64/)
2. sync the repo

Actual results:

The sync will fail with an error similar to the following:

Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384) command document too large
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384) Traceback (most recent call last):
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 224, in run
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)     self.update_content(metadata_files, url)
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 488, in update_content
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)     self.download_rpms(metadata_files, rpms_to_download, url)
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 685, in download_rpms
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)     unit = self.add_rpm_unit(metadata_files, unit)
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 632, in add_rpm_unit
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)     unit.save()
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)   File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 324, in save
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)     object_id = collection.save(doc, **write_concern)
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 2180, in save
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)     check_keys, False, manipulate, write_concern)
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 709, in _update
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)     codec_options=self.codec_options).copy()
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)   File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 216, in command
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)     self._raise_connection_failure(error)
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)   File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 343, in _raise_connection_failure
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)     raise error
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384) DocumentTooLarge: command document too large
Dec 13 14:43:42 bye-capsule pulp: py.warnings:WARNING: (31174-60384) /usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py:804: DeprecationWarning: save is deprecated. Use insert_one or replace_one instead
Dec 13 14:43:42 bye-capsule pulp: py.warnings:WARNING: (31174-60384)   sync_result_collection.save(sync_result)
Dec 13 14:43:42 bye-capsule pulp: py.warnings:WARNING: (31174-60384)
Dec 13 14:43:42 bye-capsule pulp: pulp.server.async.tasks:INFO: Task failed : [a9d888c3-8886-44b6-abf2-64141d6912d5]
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384) Task pulp.server.managers.repo.sync.sync[a9d888c3-8886-44b6-abf2-64141d6912d5] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384) Traceback (most recent call last):
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)     R = retval = fun(*args, **kwargs)
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 488, in __call__
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)     return super(Task, self).__call__(*args, **kwargs)
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)     return super(PulpTask, self).__call__(*args, **kwargs)
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)     return self.run(*args, **kwargs)
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 810, in sync
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)     raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384) PulpExecutionException: Importer indicated a failed response
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[83065288-0fc5-4f56-9dc0-d3089e8b5162] succeeded in 0.0121840410002s: None
Dec 13 14:44:34 bye-capsule pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Dec 13 14:44:34 bye-capsule pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[659a3d93-4df1-47cd-bc69-9938359aa351]
Dec 13 14:44:34 bye-capsule pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[5974d77b-bfad-4b55-9b25-b3d9501f5c1c]
Dec 13 14:44:34 bye-capsule pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.queue_download_deferred[659a3d93-4df1-47cd-bc69-9938359aa351] succeeded in 0.041977586s: None
Dec 13 14:44:34 bye-capsule pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.download_deferred[5974d77b-bfad-4b55-9b25-b3d9501f5c1c] succeeded in 0.040159016s: None

Expected results:

No error and the content is synced.

Additional info:
Comment 4 pulp-infra@redhat.com 2016-12-13 17:44:11 UTC 
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
Comment 5 pulp-infra@redhat.com 2016-12-13 17:44:13 UTC 
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 6 pulp-infra@redhat.com 2016-12-16 00:50:56 UTC 
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
Comment 7 pulp-infra@redhat.com 2017-01-20 02:04:30 UTC 
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 10 Tanya Tereshchenko 2017-02-27 17:52:58 UTC 
Created attachment 1258173 [details]
Patch to compress RPM metadata, compatible with pulp_rpm 2.8.7
Comment 11 Tanya Tereshchenko 2017-02-27 20:28:40 UTC 
Created attachment 1258189 [details]
Updated patch to compress RPM metadata for pulp_rpm 2.8.7
Comment 12 pulp-infra@redhat.com 2017-02-28 23:05:03 UTC 
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Comment 13 pulp-infra@redhat.com 2017-02-28 23:34:49 UTC 
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.
Comment 17 Tanya Tereshchenko 2017-03-13 22:48:43 UTC 
Created attachment 1262699 [details]
Patch to compress RPM metadata with re-numbered migration for pulp_rpm 2.8.7
Comment 19 Dennis Kliban 2017-03-14 02:31:24 UTC 
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
Comment 23 pulp-infra@redhat.com 2017-03-30 02:06:53 UTC 
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 24 pulp-infra@redhat.com 2017-03-30 02:06:59 UTC 
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
Comment 25 pulp-infra@redhat.com 2017-03-30 23:06:53 UTC 
The Pulp upstream bug status is at CLOSED - COMPLETE. Updating the external tracker on this bug.
Comment 26 jcallaha 2017-04-03 18:08:22 UTC 
Verified in Satellite 6.2.9 Snap 2

Followed the steps outlined in the bz description. The repository sync'd correctly, both when in on_demand and then in immediate download modes. See attached screenshot for verification.
Comment 27 Mike McCune 2017-04-06 19:24:49 UTC 
=== HOTFIX INSTRUCTIONS FOR SATELLITE 6.2.8 ONLY ===


See the hotfix package and instructions outlined here:

https://bugzilla.redhat.com/show_bug.cgi?id=1388296#c49
Comment 28 pulp-infra@redhat.com 2017-04-19 21:37:22 UTC 
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 30 errata-xmlrpc 2017-05-01 13:56:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1191
Note You need to log in before you can comment on or make changes to this bug.