Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
First Last Prev Next    This bug is not in your last search results.
Bug 1404345 - Sync failure - DocumentTooLarge
Sync failure - DocumentTooLarge
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Pulp (Show other bugs)
6.2.4
Unspecified Unspecified
urgent Severity urgent with 2 votes (vote)
: 6.2.9
: Unused
Assigned To: satellite6-bugs
jcallaha
: PrioBumpField, PrioBumpPM, Triaged
Depends On:
Blocks:
  Show dependency tree / graph
 
Reported: 2016-12-13 11:03 EST by Brad Buckingham
Modified: 2018-09-19 11:08 EDT (History)
19 users (show)

See Also:
Fixed In Version: pulp-rpm-2.8.7.9-1
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-05-01 09:56:59 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Patch to compress RPM metadata, compatible with pulp_rpm 2.8.7 (26.51 KB, patch)
2017-02-27 12:52 EST, Tanya Tereshchenko 		no flags Details | Diff
Updated patch to compress RPM metadata for pulp_rpm 2.8.7 (27.79 KB, patch)
2017-02-27 15:28 EST, Tanya Tereshchenko 		no flags Details | Diff
Patch to compress RPM metadata with re-numbered migration for pulp_rpm 2.8.7 (26.69 KB, patch)
2017-03-13 18:48 EDT, Tanya Tereshchenko 		no flags Details | Diff
Show Obsolete (2) Add an attachment (proposed patch, testcase, etc.)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2957371 None None None 2017-03-07 10:10 EST
Pulp Redmine 2652 Normal CLOSED - COMPLETE Backport #723, RPMs with large number of files can exceed mongo document size limit 2017-03-30 19:06 EDT
Pulp Redmine 723 High CLOSED - CURRENTRELEASE RPMs with large number of files can exceed mongo document size limit 2017-04-19 17:37 EDT
Red Hat Product Errata RHBA-2017:1191 normal SHIPPED_LIVE Satellite 6.2.9 Async Bug Release 2017-05-01 13:49:42 EDT

  None (edit)
Description Brad Buckingham 2016-12-13 11:03:02 EST 
Description of problem:

Unable to sync a repository that has RPMs that are 'too big' (e.g. 1+ GB).

Version-Release number of selected component (if applicable):

Satellite 6.2

How reproducible:

Always, if the RPM is over the limit

Steps to Reproduce:

1. create a repo (e.g. https://fedorapeople.org/groups/katello/releases/yum/foreman/1.13/el6/x86_64/)
2. sync the repo

Actual results:

The sync will fail with an error similar to the following:

Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384) command document too large
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384) Traceback (most recent call last):
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 224, in run
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)     self.update_content(metadata_files, url)
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 488, in update_content
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)     self.download_rpms(metadata_files, rpms_to_download, url)
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 685, in download_rpms
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)     unit = self.add_rpm_unit(metadata_files, unit)
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 632, in add_rpm_unit
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)     unit.save()
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)   File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 324, in save
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)     object_id = collection.save(doc, **write_concern)
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 2180, in save
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)     check_keys, False, manipulate, write_concern)
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 709, in _update
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)     codec_options=self.codec_options).copy()
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)   File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 216, in command
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)     self._raise_connection_failure(error)
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)   File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 343, in _raise_connection_failure
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384)     raise error
Dec 13 14:43:42 bye-capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (31174-60384) DocumentTooLarge: command document too large
Dec 13 14:43:42 bye-capsule pulp: py.warnings:WARNING: (31174-60384) /usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py:804: DeprecationWarning: save is deprecated. Use insert_one or replace_one instead
Dec 13 14:43:42 bye-capsule pulp: py.warnings:WARNING: (31174-60384)   sync_result_collection.save(sync_result)
Dec 13 14:43:42 bye-capsule pulp: py.warnings:WARNING: (31174-60384)
Dec 13 14:43:42 bye-capsule pulp: pulp.server.async.tasks:INFO: Task failed : [a9d888c3-8886-44b6-abf2-64141d6912d5]
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384) Task pulp.server.managers.repo.sync.sync[a9d888c3-8886-44b6-abf2-64141d6912d5] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384) Traceback (most recent call last):
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)     R = retval = fun(*args, **kwargs)
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 488, in __call__
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)     return super(Task, self).__call__(*args, **kwargs)
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)     return super(PulpTask, self).__call__(*args, **kwargs)
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)     return self.run(*args, **kwargs)
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 810, in sync
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384)     raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:ERROR: (30789-60384) PulpExecutionException: Importer indicated a failed response
Dec 13 14:43:43 bye-capsule pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[83065288-0fc5-4f56-9dc0-d3089e8b5162] succeeded in 0.0121840410002s: None
Dec 13 14:44:34 bye-capsule pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Dec 13 14:44:34 bye-capsule pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[659a3d93-4df1-47cd-bc69-9938359aa351]
Dec 13 14:44:34 bye-capsule pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[5974d77b-bfad-4b55-9b25-b3d9501f5c1c]
Dec 13 14:44:34 bye-capsule pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.queue_download_deferred[659a3d93-4df1-47cd-bc69-9938359aa351] succeeded in 0.041977586s: None
Dec 13 14:44:34 bye-capsule pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.download_deferred[5974d77b-bfad-4b55-9b25-b3d9501f5c1c] succeeded in 0.040159016s: None

Expected results:

No error and the content is synced.

Additional info:
Comment 4 pulp-infra@redhat.com 2016-12-13 12:44:11 EST 
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
Comment 5 pulp-infra@redhat.com 2016-12-13 12:44:13 EST 
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 6 pulp-infra@redhat.com 2016-12-15 19:50:56 EST 
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
Comment 7 pulp-infra@redhat.com 2017-01-19 21:04:30 EST 
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 10 Tanya Tereshchenko 2017-02-27 12:52 EST 
Created attachment 1258173 [details]
Patch to compress RPM metadata, compatible with pulp_rpm 2.8.7
Comment 11 Tanya Tereshchenko 2017-02-27 15:28 EST 
Created attachment 1258189 [details]
Updated patch to compress RPM metadata for pulp_rpm 2.8.7
Comment 12 pulp-infra@redhat.com 2017-02-28 18:05:03 EST 
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Comment 13 pulp-infra@redhat.com 2017-02-28 18:34:49 EST 
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.
Comment 17 Tanya Tereshchenko 2017-03-13 18:48 EDT 
Created attachment 1262699 [details]
Patch to compress RPM metadata with re-numbered migration for pulp_rpm 2.8.7
Comment 19 Dennis Kliban 2017-03-13 22:31:24 EDT 
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
Comment 23 pulp-infra@redhat.com 2017-03-29 22:06:53 EDT 
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 24 pulp-infra@redhat.com 2017-03-29 22:06:59 EDT 
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
Comment 25 pulp-infra@redhat.com 2017-03-30 19:06:53 EDT 
The Pulp upstream bug status is at CLOSED - COMPLETE. Updating the external tracker on this bug.
Comment 26 jcallaha 2017-04-03 14:08:22 EDT 
Verified in Satellite 6.2.9 Snap 2

Followed the steps outlined in the bz description. The repository sync'd correctly, both when in on_demand and then in immediate download modes. See attached screenshot for verification.
Comment 27 Mike McCune 2017-04-06 15:24:49 EDT 
=== HOTFIX INSTRUCTIONS FOR SATELLITE 6.2.8 ONLY ===


See the hotfix package and instructions outlined here:

https://bugzilla.redhat.com/show_bug.cgi?id=1388296#c49
Comment 28 pulp-infra@redhat.com 2017-04-19 17:37:22 EDT 
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 30 errata-xmlrpc 2017-05-01 09:56:59 EDT 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1191
Note You need to log in before you can comment on or make changes to this bug.
First Last Prev Next    This bug is not in your last search results.