Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1405835

Summary: synchronizing a repository fails in "PLP0000: Importer indicated a failed response" due to timeouted cursor in _duplicate_key_id_generator_aggregation
Product: Red Hat Satellite Reporter: Pavel Moravec <pmoravec>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Roman Plevka <rplevka>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.2.5CC: bbuckingham, bkearney, bmbouter, dalley, daviddavis, dkliban, ggainey, idyurev, ipanova, jcallaha, mhrivnak, oshtaier, pcreech, rchan, rplevka, ttereshc, zhunting
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: pulp-rpm-2.8.7.13-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1446731 (view as bug list) Environment:
Last Closed: 2017-06-20 17:53:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pavel Moravec 2016-12-18 19:44:39 UTC 
Description of problem:
Actions::Katello::ContentView::CapsuleGenerateAndSync failed on step Actions::Pulp::Consumer::SyncCapsule i.e. on pulp task pulp.server.managers.repo.sync.sync trying to synchronize a repo from Satellite to Capsule. Error string:

PulpExecutionException: Importer indicated a failed response

/var/log/messages has the reasoning behind:

Dec 15 19:23:38 capsule pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[d12f21ca-23b0-42e4-8270-ae405ec848f4]
..
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) Cursor not found, cursor id: 159997743246
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) Traceback (most recent call last):
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 288, in run
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)     purge.remove_repo_duplicate_nevra(self.conduit.repo_id)
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 304, in remove_repo_duplicate_nevra
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)     for unit_ids in _duplicate_key_id_generator(unit_type):
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 403, in _duplicate_key_id_generator_aggregation
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)     for pkg in aggregation:
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)   File "/usr/lib64/python2.7/site-packages/pymongo/command_cursor.py", line 232, in next
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)     if len(self.__data) or self._refresh():
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)   File "/usr/lib64/python2.7/site-packages/pymongo/command_cursor.py", line 190, in _refresh
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)     self.__collection.codec_options))
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)   File "/usr/lib64/python2.7/site-packages/pymongo/command_cursor.py", line 116, in __send_message
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)     self.__collection.codec_options)
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)   File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 114, in _unpack_response
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)     raise CursorNotFound(msg, 43, errobj)
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) CursorNotFound: Cursor not found, cursor id: 159997743246
Dec 15 21:19:54 ux0501 pulp: pulp.server.async.tasks:INFO: Task failed : [d12f21ca-23b0-42e4-8270-ae405ec848f4]
..
Dec 15 21:20:02 ux0501 pulp: celery.worker.job:ERROR: (25306-78624) Task pulp.server.managers.repo.sync.sync[d12f21ca-23b0-42e4-8270-ae405ec848f4] raised unexpected: PulpExecutionException('Importer indicated a failed response',)

I.e. in source code:
/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py :

def _duplicate_key_id_generator_aggregation(unit, fields):
..
    # When aggregating over hundreds of thousands of packages, mongo can overflow
    # To prevent this, mongo needs to be allowed to temporarily use the disk for this transaction
    aggregation = unit.objects.aggregate(sort, project, allowDiskUse=True)

    # loop state tracking vars
    previous_nevra = None
    previous_pkg_id = None
    yielding_ids = None

    for pkg in aggregation:
..

the aggregation timeouts on its cursor.


Version-Release number of selected component (if applicable):
pulp-server-2.8.7.3-1.el7sat.noarch


How reproducible:
???


Steps to Reproduce:
1. Have several large repos being synced in parallel (doesnt matter if from CDN to Sat or from Sat to Caps, I guess)


Actual results:
See above failure of the pulp sync task


Expected results:
No failed task


Additional info:
I expect:

aggregate(sort, project, allowDiskUse=True).batch_size(10)

is a workaround (possibly with different batch size value).
Comment 3 pulp-infra@redhat.com 2016-12-20 19:31:23 UTC 
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
Comment 4 pulp-infra@redhat.com 2016-12-20 19:31:25 UTC 
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
Comment 5 pulp-infra@redhat.com 2017-01-03 16:32:10 UTC 
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 6 pulp-infra@redhat.com 2017-01-04 12:01:52 UTC 
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
Comment 7 Michael Hrivnak 2017-01-11 19:45:58 UTC 
*** Bug 1401636 has been marked as a duplicate of this bug. ***
Comment 8 pulp-infra@redhat.com 2017-01-16 14:31:32 UTC 
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 9 pulp-infra@redhat.com 2017-01-30 18:01:35 UTC 
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Comment 10 pulp-infra@redhat.com 2017-01-30 18:31:54 UTC 
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.
Comment 11 pulp-infra@redhat.com 2017-02-16 20:32:26 UTC 
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
Comment 12 pulp-infra@redhat.com 2017-02-23 21:02:03 UTC 
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 13 pulp-infra@redhat.com 2017-04-17 18:33:09 UTC 
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
Comment 14 pulp-infra@redhat.com 2017-04-17 18:33:12 UTC 
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
Comment 15 Daniel Alley 2017-04-17 20:42:32 UTC 
Remove external tracker for backport task - breaks some automation
Comment 16 Roman Plevka 2017-06-06 13:39:51 UTC 
VERIFIED
on sat6.2.10-2
was unable to reproduce this while syncing rhel5|6|7 server repos simultaneously
Comment 17 Bryan Kearney 2017-06-20 17:53:09 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1553