Description of problem: syncing docker-type repos with registry url set to "registry.docker.io" (as the help message suggests) no longer works. Pulp returns a cryptic traceback where one is unable to tell the root cause of the failed sync: PLP0000: tuple index out of range traceback: | Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task R = retval = fun(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in __call__ return super(Task, self).__call__(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in __call__ return super(PulpTask, self).__call__(*args, **kwargs) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__ return self.run(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 796, in sync sync_report = sync_repo(transfer_repo, conduit, call_config) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 892, in wrap_f return f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 86, in sync_repo return self.sync_step.process_lifecycle() File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle super(PluginStep, self).process_lifecycle() File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle step.process() File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 257, in process self._process_block() File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 304, in _process_block self.process_main() File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py", line 235, in process_main self._process_manifest_list(manifest, digest, available_blobs, tag) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py", line 272, in _process_manifest_list tag=False) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/registry.py", line 388, in get_manifest response_headers, manifest = self._get_path(path, headers=request_headers) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/registry.py", line 497, in _get_path report = self.downloader.download_one(request) File "/usr/lib/python2.7/site-packages/nectar/downloaders/base.py", line 68, in download_one return self._download_one(request) File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 254, in _download_one return self._fetch(request, session) File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 378, in _fetch if nretry < DEFAULT_GENERIC_TRIES - 1 and e.args[1] == errno.ECONNRESET: IndexError: tuple index out of range Looks like the "e"xception args tuple doesn't contain more than one element. I guess the dockerhubs registry url changed. I tried to use `docker -D pull busybox` to see what url my docker client uses and found this: docker.io/library/busybox:latest however, changing the registry url to this makes katello automatically reach out to https://docker.io/v2/library/busybox/tags/list where it receives 404 Version-Release number of selected component (if applicable): 6.9.0-1 Steps to Reproduce: 1. try to create a custom repository with docker content-type, using the registry.docker.io as upstream url 2. use `busybox` image 3. try to sync Actual results: see above
> I guess the dockerhubs registry url changed. My bet would be on rate limiting they started to enforce earlier this month: https://www.docker.com/blog/what-you-need-to-know-about-upcoming-docker-hub-rate-limiting/ We need someone to confirm this, but if this is the case, catching exception and displaying some user-friendly error message is about the only thing we can do.
Yes, i believe this is a result of the rate limiting as well. Under pulp3 the error ends up being 'too many requests'. Moving this to the pulp component, as that's where the error handling needs improvement.
I have checked the snapshot and it contains python-nectar-1.6.2-1.el7sat.noarch.rpm instead of 1.6.4. The snap should be updated to the latest nectar version which contains the fix for the opened BZ. @egolov PTAL
Apparently the latest snap alyready contains correct nectar version.We should close out this BZ.
yes, we have 1.6.4, so we should be good
Steps to test: 1. Create a custom prod 2. Create a custom docker repo with https://quay.io/ and upstream repo as quay/busybox 3. Sync 4. Check sync status 5. Create another customer docker repo with https://registry.redhat.io and upstream repo openshift3/mediawiki-apb 6. Sync Expected result Should successfully syncs repo Actual result: Successfully syncs repo Note: we're using a different url since docker.io has enforced limits. Verified on 6.9.0_09
Created attachment 1749145 [details] Results on docker sync
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Satellite 6.9 Release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:1313