Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1356111

Summary: [upgrade] repository.distributor_update task failed with error: KeyError(u'repo-registry-id',)
Product: Red Hat Satellite Reporter: Sachin Ghai <sghai>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Sachin Ghai <sghai>
Severity: medium Docs Contact:
Priority: unspecified    
Version: UnspecifiedCC: bbuckingham, bmbouter, daviddavis, dkliban, ggainey, ipanova, jcallaha, jsherril, mhrivnak, pcreech, rchan, ttereshc
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-21 17:00:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
complete exception none

Description Sachin Ghai 2016-07-13 12:37:26 UTC 
Description of problem:
I got following error in logs while performing satellite upgrade from sat6.1.9 -> sat6.2 GA snap19.1 on rhel7


Jul 13 08:11:20 qe-sat6-upgrade-rhel7 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[be1c0cd1-8d5a-4fde-8241-ebf446fe0bfb]
Jul 13 08:11:20 qe-sat6-upgrade-rhel7 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[306a687b-a1d4-4ded-a897-0993bebd86d0]
Jul 13 08:11:20 qe-sat6-upgrade-rhel7 pulp: celery.worker.strategy:INFO: Received task: pulp.server.tasks.repository.distributor_update[8bd224ce-1ff8-478d-beae-34762d45e959]
Jul 13 08:11:20 qe-sat6-upgrade-rhel7 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[1452caf6-5d62-4b47-b01c-cf7b84494104]
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[be1c0cd1-8d5a-4fde-8241-ebf446fe0bfb] succeeded in 0.0651753509883s: None
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: pulp.server.async.tasks:INFO: Task failed : [8bd224ce-1ff8-478d-beae-34762d45e959]
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.importer.update_importer_config[858e876e-d395-4df0-b610-f1b0e3d49690]
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[3f4e0b38-96f2-47ee-8cad-2441bb970147]
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[306a687b-a1d4-4ded-a897-0993bebd86d0] succeeded in 0.0630636169954s: None
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584) Task pulp.server.tasks.repository.distributor_update[8bd224ce-1ff8-478d-beae-34762d45e959] raised unexpected: KeyError(u'repo-registry-id',)
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584) Traceback (most recent call last):
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584)     R = retval = fun(*args, **kwargs)
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584)     return super(Task, self).__call__(*args, **kwargs)
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584)     return super(PulpTask, self).__call__(*args, **kwargs)
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584)     return self.run(*args, **kwargs)
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/distributor.py", line 211, in update
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584)     distributor.config.pop(k)
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584)   File "/usr/lib/python2.7/site-packages/mongoengine/base/datastructures.py", line 73, in pop
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584)     return super(BaseDict, self).pop(*args, **kwargs)
Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584) KeyError: u'repo-registry-id'


Please note that this exception was raised when installer was at step:
Upgrade Step: add_export_distributor (this may take a while) ...


And installer was completed successfully

Version-Release number of selected component (if applicable):
upgrade from sat6.1.9 -> sat6.2 GA snap19.1 on rhel7


How reproducible:


Steps to Reproduce:
1. set sat6.2 repo
2. katello-service stop
3. yum update -y
4. satellite-installer --scenario satellite --upgrade

Actual results:


Expected results:
no exception should be raised in logs when installer completed successfully

Additional info:
Comment 1 Sachin Ghai 2016-07-13 12:41:46 UTC 
Created attachment 1179238 [details]
complete exception
Comment 2 Brad Buckingham 2016-07-13 13:18:01 UTC 
Michael, any thoughts on what may cause the above exception and the impacts it may have?
Comment 3 Justin Sherrill 2016-07-13 13:28:51 UTC 
This task 'add_export_distributor' is attempting to add an export distributor to our yum repos.  As part of that process it does a full 'refresh' of the repository, updating the importers and distributors of all repos if they are not exactly what we expect.  

Judging by the traceback:

File "/usr/lib/python2.7/site-packages/pulp/server/controllers/distributor.py", line 211, in update

I'm guessing this was a distributor update, and by the error containing  'repo-registry-id' it was a docker repo.
Comment 4 Michael Hrivnak 2016-07-13 17:34:02 UTC 
Here is what I think is going on, but it would be helpful to have feedback from the katello side on whether it sounds right.

It appears that the distributor config is being updated, and that in the new config object being provided, the key "repo-registry-id" has a value of None. Here is what the REST API docs say about values of None in that case:

"any value that is set to explicitly to None will be removed from the config."

It appears that the corresponding code in pulp assumes that if a key has a value of None in the new config, it must be present in the old config, and then tries to remove it.

Apparently in this case, katello includes "repo-registry-id: None" in the new config, but the old one didn't contain that key, so it is resulting in an unhandled exception.

Ideally pulp should tolerate this condition. I'll file an upstream bug on it.

Do you think I'm right about what katello is doing? Is katello deliberately trying to remove the repo-registry-id setting from distributor configs?
Comment 6 Justin Sherrill 2016-07-13 18:52:38 UTC 
Michael, It does seem that that is the problem.   The odd thing is that pulp does not return any error at all, just logs the exception (so this bz isn't really fatal at all). 

It seems we always passing repo-registry-id as None no matter what on creation and update.  I'm not sure why, but it was added explicitly at some point.
Comment 9 pulp-infra@redhat.com 2016-11-21 18:52:44 UTC 
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
Comment 10 pulp-infra@redhat.com 2016-11-21 18:52:47 UTC 
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
Comment 11 pulp-infra@redhat.com 2016-12-12 17:32:04 UTC 
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 12 pulp-infra@redhat.com 2016-12-12 17:32:07 UTC 
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 13 pulp-infra@redhat.com 2016-12-12 18:01:52 UTC 
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.
Comment 15 Sachin Ghai 2017-11-17 11:41:18 UTC 
Verified with upgrade from 6.2.12 -> sat6.3 snap24. The reported error KeyError: u'repo-registry-id' no longer reproducible. Moving to verified.
Comment 16 Bryan Kearney 2018-02-21 17:00:35 UTC 
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0336