1323730 – [lazy sync] Exporting RHEL 7Server RPMs repository fails with BSON document too large

Bug 1323730 - [lazy sync] Exporting RHEL 7Server RPMs repository fails with BSON document too large

Summary: [lazy sync] Exporting RHEL 7Server RPMs repository fails with BSON document t...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Satellite 6
Classification:	Red Hat
Component:	Pulp
Sub Component:
Version:	6.2.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high vote
Target Milestone:	Unspecified
Assignee:	satellite6-bugs
QA Contact:	Sanket Jagtap
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1707778
TreeView+	depends on / blocked

Reported:	2016-04-04 14:24 UTC by Og Maciel
Modified:	2019-09-26 17:34 UTC (History)
CC List:	12 users (show)
Fixed In Version:	pulp-rpm-2.8.7.2-1 pulp-2.8.7.1-1
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Clones:	1707778 (view as bug list)
Environment:
Last Closed:	2016-10-26 12:25:49 UTC
Target Upstream Version:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Links
System	ID	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2016:2108	normal	SHIPPED_LIVE	Satellite 6.2.3 Async Bug Release	2016-10-26 16:21:52 UTC
Pulp Redmine	1808	Normal	CLOSED - CURRENTRELEASE	exporting a sufficiently large repo with 'on_demand' policy results in BSON error	2016-05-17 20:00:57 UTC
Pulp Redmine	1835	High	CLOSED - CURRENTRELEASE	export fails when units are not downloaded	2016-05-17 20:00:39 UTC

Description Og Maciel 2016-04-04 14:24:57 UTC

Description of problem:

While attempting to verify https://bugzilla.redhat.com/show_bug.cgi?id=1312640 I tried to export "Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server" which failed with the generic error: uninitialized constant Actions::Pulp::AbstractAsyncTask::PulpError

Version-Release number of selected component (if applicable):

* Satellite 6.2.0 SNAP 6.1

How reproducible:


Steps to Reproduce:
1. Synchronize "Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server" 
2. Try to export it with hammer repository export command
3.

Actual results:


Expected results:


Additional info:


Apr  4 10:16:38 ibm-x3250m4-01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[24873ade-93e8-433c-875a-79da03a44031] succeeded in 0.0370843109849s: None
Apr  4 10:18:44 ibm-x3250m4-01 systemd-logind: New session 255 of user root.
Apr  4 10:18:44 ibm-x3250m4-01 systemd: Started Session 255 of user root.
Apr  4 10:18:44 ibm-x3250m4-01 systemd: Starting Session 255 of user root.
Apr  4 10:19:08 ibm-x3250m4-01 pulp: py.warnings:WARNING: (11630-68832) /usr/lib/python2.7/site-packages/pulp/server/managers/repo/group/publish.py:79: DeprecationWarning: save is deprecated. Use insert_one or replace_one instead
Apr  4 10:19:08 ibm-x3250m4-01 pulp: py.warnings:WARNING: (11630-68832)   distributor_coll.save(distributor)
Apr  4 10:19:08 ibm-x3250m4-01 pulp: py.warnings:WARNING: (11630-68832)
Apr  4 10:19:08 ibm-x3250m4-01 pulp: py.warnings:WARNING: (11630-68832) /usr/lib/python2.7/site-packages/pulp/server/managers/repo/group/publish.py:85: DeprecationWarning: save is deprecated. Use insert_one or replace_one instead
Apr  4 10:19:08 ibm-x3250m4-01 pulp: py.warnings:WARNING: (11630-68832)   publish_result_coll.save(result)
Apr  4 10:19:08 ibm-x3250m4-01 pulp: py.warnings:WARNING: (11630-68832)
Apr  4 10:19:08 ibm-x3250m4-01 pulp: pulp.server.async.tasks:INFO: Task failed : [1a09ba40-1278-45a0-8098-645faaaab693]
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832) Task pulp.server.managers.repo.group.publish.publish[1a09ba40-1278-45a0-8098-645faaaab693] raised unexpected: DocumentTooLarge('BSON document too large (18598541 bytes) - the connected serversupports BSON document sizes up to 16777216 bytes.',)
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832) Traceback (most recent call last):
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)     R = retval = fun(*args, **kwargs)
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)     return super(Task, self).__call__(*args, **kwargs)
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)     return super(PulpTask, self).__call__(*args, **kwargs)
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)     return self.run(*args, **kwargs)
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/group/publish.py", line 59, in publish
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)     conduit, call_config)
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/group/publish.py", line 85, in _do_publish
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)     publish_result_coll.save(result)
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 2180, in save
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)     check_keys, False, manipulate, write_concern)
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 728, in _update
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)     self.codec_options)
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 458, in _legacy_write
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)     rqst_id, msg, max_size, acknowledged)
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)   File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 261, in legacy_write
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)     self.send_message(msg, max_doc_size)
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)   File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 228, in send_message
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832)     (max_doc_size, self.max_bson_size))
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:ERROR: (11392-68832) DocumentTooLarge: BSON document too large (18598541 bytes) - the connected serversupports BSON document sizes up to 16777216 bytes.
Apr  4 10:19:09 ibm-x3250m4-01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[43032b08-2775-404f-b9e6-a4e2056f5f44] succeeded in 0.00403834600002s: None
Apr  4 10:19:19 ibm-x3250m4-01 pulp: py.warnings:WARNING: (17854-22400) /usr/lib/python2.7/site-packages/pulp/server/managers/repo/group/distributor.py:230: DeprecationWarning: remove is deprecated. Use delete_one or delete_many instead.
Apr  4 10:19:19 ibm-x3250m4-01 pulp: py.warnings:WARNING: (17854-22400)   distributor_coll.remove(distributor)
Apr  4 10:19:19 ibm-x3250m4-01 pulp: py.warnings:WARNING: (17854-22400)
Apr  4 10:19:19 ibm-x3250m4-01 pulp: py.warnings:WARNING: (17854-22400) /usr/lib/python2.7/site-packages/pulp/server/managers/repo/group/cud.py:168: DeprecationWarning: remove is deprecated. Use delete_one or delete_many instead.
Apr  4 10:19:19 ibm-x3250m4-01 pulp: py.warnings:WARNING: (17854-22400)   collection.remove({'id': group_id})
Apr  4 10:19:19 ibm-x3250m4-01 pulp: py.warnings:WARNING: (17854-22400)

Comment 2 pulp-infra@redhat.com 2016-04-04 15:03:39 UTC

The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 3 pulp-infra@redhat.com 2016-04-04 15:03:41 UTC

The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 5 pulp-infra@redhat.com 2016-04-08 21:03:41 UTC

The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 6 pulp-infra@redhat.com 2016-04-08 21:33:41 UTC

The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 7 pulp-infra@redhat.com 2016-04-11 14:03:39 UTC

The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 8 pulp-infra@redhat.com 2016-04-11 14:03:42 UTC

The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 9 pulp-infra@redhat.com 2016-04-11 16:03:44 UTC

The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 10 pulp-infra@redhat.com 2016-04-14 14:03:48 UTC

The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 11 pulp-infra@redhat.com 2016-04-15 13:33:36 UTC

The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 12 pulp-infra@redhat.com 2016-04-15 15:33:35 UTC

The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 13 pulp-infra@redhat.com 2016-04-18 19:33:40 UTC

The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 14 Brad Buckingham 2016-04-21 19:34:41 UTC

WORKAROUND:

Configure the repositories being exported with a download policy of 'immediate'.  Note: This is the policy used in the Satellite 6.2 Beta.

Comment 15 pulp-infra@redhat.com 2016-04-26 23:03:47 UTC

The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 16 pulp-infra@redhat.com 2016-04-26 23:04:03 UTC

The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 17 pulp-infra@redhat.com 2016-05-06 17:20:50 UTC

The Pulp upstream bug status is at VERIFIED. Updating the external tracker on this bug.

Comment 18 Mike McCune 2016-05-16 05:39:57 UTC

This is a lazy-sync related bug, moving to 6.2.z

Comment 19 pulp-infra@redhat.com 2016-05-17 20:00:40 UTC

The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 20 pulp-infra@redhat.com 2016-05-17 20:00:58 UTC

The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 22 Brad Buckingham 2016-07-25 13:21:22 UTC

Moving to POST as upstream fix is available.

Comment 23 Sanket Jagtap 2016-10-14 06:33:01 UTC

Build: Satellite 6.2.3 snap 2.0

Verification Steps:
1. Created a custom product and set the download policy to on-demand
2. Tried to export the product

hammer> repository info --id 5 --organization-id 1
ID:                 5
Name:               The_product_repo
Label:              The_product_repo
Organization:       Default Organization
Red Hat Repository: no
Content Type:       yum
Checksum Type:      sha256
URL:                http://abc.redhat.com/pulp/repos/Sat6-CI/QA/Satellite_
RHEL7/custom/Red_Hat_Satellite_6_2_Composes/RHEL7_Satellite_x86_64_os/
Publish Via HTTP:   yes
Published At:       http://xyz.redhat.com/pulp/repos/Default_Organization/
Library/custom/The_product/The_product_repo/
Relative Path:      Default_Organization/Library/custom/The_product/The_
product_repo
*Download Policy:    on_demand*
Product:
    ID:   7
    Name: The_product
GPG Key:

Sync:
    Status:         Success
    Last Sync Date: 1 minute
Created:            2016/10/12 06:27:48
Updated:            2016/10/12 06:28:48
Content Counts:
    Packages:       302
    Package Groups: 1
    Errata:         0

hammer> repository export --id 5
[..........................                                             ]
[37%]
Error: PLP0046: The repository group cannot be exported because these
repos have units that are not downloaded: Default_Organization-The_
product-The_product_repo

The task has been started and then failed at 37% , as
expected with proper error.

Marking this to verified.

Comment 26 errata-xmlrpc 2016-10-26 12:25:49 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2108

Note You need to log in before you can comment on or make changes to this bug.