Description of problem: I got following error in logs while performing satellite upgrade from sat6.1.9 -> sat6.2 GA snap19.1 on rhel7 Jul 13 08:11:20 qe-sat6-upgrade-rhel7 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[be1c0cd1-8d5a-4fde-8241-ebf446fe0bfb] Jul 13 08:11:20 qe-sat6-upgrade-rhel7 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[306a687b-a1d4-4ded-a897-0993bebd86d0] Jul 13 08:11:20 qe-sat6-upgrade-rhel7 pulp: celery.worker.strategy:INFO: Received task: pulp.server.tasks.repository.distributor_update[8bd224ce-1ff8-478d-beae-34762d45e959] Jul 13 08:11:20 qe-sat6-upgrade-rhel7 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[1452caf6-5d62-4b47-b01c-cf7b84494104] Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[be1c0cd1-8d5a-4fde-8241-ebf446fe0bfb] succeeded in 0.0651753509883s: None Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: pulp.server.async.tasks:INFO: Task failed : [8bd224ce-1ff8-478d-beae-34762d45e959] Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.importer.update_importer_config[858e876e-d395-4df0-b610-f1b0e3d49690] Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[3f4e0b38-96f2-47ee-8cad-2441bb970147] Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[306a687b-a1d4-4ded-a897-0993bebd86d0] succeeded in 0.0630636169954s: None Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584) Task pulp.server.tasks.repository.distributor_update[8bd224ce-1ff8-478d-beae-34762d45e959] raised unexpected: KeyError(u'repo-registry-id',) Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584) Traceback (most recent call last): Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584) R = retval = fun(*args, **kwargs) Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__ Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584) return super(Task, self).__call__(*args, **kwargs) Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__ Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584) return super(PulpTask, self).__call__(*args, **kwargs) Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__ Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584) return self.run(*args, **kwargs) Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/distributor.py", line 211, in update Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584) distributor.config.pop(k) Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584) File "/usr/lib/python2.7/site-packages/mongoengine/base/datastructures.py", line 73, in pop Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584) return super(BaseDict, self).pop(*args, **kwargs) Jul 13 08:11:21 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (22429-15584) KeyError: u'repo-registry-id' Please note that this exception was raised when installer was at step: Upgrade Step: add_export_distributor (this may take a while) ... And installer was completed successfully Version-Release number of selected component (if applicable): upgrade from sat6.1.9 -> sat6.2 GA snap19.1 on rhel7 How reproducible: Steps to Reproduce: 1. set sat6.2 repo 2. katello-service stop 3. yum update -y 4. satellite-installer --scenario satellite --upgrade Actual results: Expected results: no exception should be raised in logs when installer completed successfully Additional info:
Created attachment 1179238 [details] complete exception
Michael, any thoughts on what may cause the above exception and the impacts it may have?
This task 'add_export_distributor' is attempting to add an export distributor to our yum repos. As part of that process it does a full 'refresh' of the repository, updating the importers and distributors of all repos if they are not exactly what we expect. Judging by the traceback: File "/usr/lib/python2.7/site-packages/pulp/server/controllers/distributor.py", line 211, in update I'm guessing this was a distributor update, and by the error containing 'repo-registry-id' it was a docker repo.
Here is what I think is going on, but it would be helpful to have feedback from the katello side on whether it sounds right. It appears that the distributor config is being updated, and that in the new config object being provided, the key "repo-registry-id" has a value of None. Here is what the REST API docs say about values of None in that case: "any value that is set to explicitly to None will be removed from the config." It appears that the corresponding code in pulp assumes that if a key has a value of None in the new config, it must be present in the old config, and then tries to remove it. Apparently in this case, katello includes "repo-registry-id: None" in the new config, but the old one didn't contain that key, so it is resulting in an unhandled exception. Ideally pulp should tolerate this condition. I'll file an upstream bug on it. Do you think I'm right about what katello is doing? Is katello deliberately trying to remove the repo-registry-id setting from distributor configs?
Michael, It does seem that that is the problem. The odd thing is that pulp does not return any error at all, just logs the exception (so this bz isn't really fatal at all). It seems we always passing repo-registry-id as None no matter what on creation and update. I'm not sure why, but it was added explicitly at some point.
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.
Verified with upgrade from 6.2.12 -> sat6.3 snap24. The reported error KeyError: u'repo-registry-id' no longer reproducible. Moving to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:0336