Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
syncing docker-type repos with registry url set to "registry.docker.io" (as the help message suggests) no longer works.
Pulp returns a cryptic traceback where one is unable to tell the root cause of the failed sync:
PLP0000: tuple index out of range
traceback: |
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in __call__
return super(Task, self).__call__(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in __call__
return super(PulpTask, self).__call__(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 796, in sync
sync_report = sync_repo(transfer_repo, conduit, call_config)
File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 892, in wrap_f
return f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 86, in sync_repo
return self.sync_step.process_lifecycle()
File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle
super(PluginStep, self).process_lifecycle()
File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle
step.process()
File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 257, in process
self._process_block()
File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 304, in _process_block
self.process_main()
File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py", line 235, in process_main
self._process_manifest_list(manifest, digest, available_blobs, tag)
File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py", line 272, in _process_manifest_list
tag=False)
File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/registry.py", line 388, in get_manifest
response_headers, manifest = self._get_path(path, headers=request_headers)
File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/registry.py", line 497, in _get_path
report = self.downloader.download_one(request)
File "/usr/lib/python2.7/site-packages/nectar/downloaders/base.py", line 68, in download_one
return self._download_one(request)
File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 254, in _download_one
return self._fetch(request, session)
File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 378, in _fetch
if nretry < DEFAULT_GENERIC_TRIES - 1 and e.args[1] == errno.ECONNRESET:
IndexError: tuple index out of range
Looks like the "e"xception args tuple doesn't contain more than one element.
I guess the dockerhubs registry url changed.
I tried to use `docker -D pull busybox` to see what url my docker client uses and found this:
docker.io/library/busybox:latest
however, changing the registry url to this makes katello automatically reach out to https://docker.io/v2/library/busybox/tags/list
where it receives 404
Version-Release number of selected component (if applicable):
6.9.0-1
Steps to Reproduce:
1. try to create a custom repository with docker content-type, using the registry.docker.io as upstream url
2. use `busybox` image
3. try to sync
Actual results:
see above
> I guess the dockerhubs registry url changed.
My bet would be on rate limiting they started to enforce earlier this month:
https://www.docker.com/blog/what-you-need-to-know-about-upcoming-docker-hub-rate-limiting/
We need someone to confirm this, but if this is the case, catching exception and displaying some user-friendly error message is about the only thing we can do.
Yes, i believe this is a result of the rate limiting as well. Under pulp3 the error ends up being 'too many requests'.
Moving this to the pulp component, as that's where the error handling needs improvement.
I have checked the snapshot and it contains python-nectar-1.6.2-1.el7sat.noarch.rpm instead of 1.6.4. The snap should be updated to the latest nectar version which contains the fix for the opened BZ.
@egolov PTAL
Steps to test:
1. Create a custom prod
2. Create a custom docker repo with https://quay.io/ and upstream repo as quay/busybox
3. Sync
4. Check sync status
5. Create another customer docker repo with https://registry.redhat.io and upstream repo openshift3/mediawiki-apb
6. Sync
Expected result
Should successfully syncs repo
Actual result:
Successfully syncs repo
Note: we're using a different url since docker.io has enforced limits.
Verified on 6.9.0_09
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Moderate: Satellite 6.9 Release), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2021:1313
Description of problem: syncing docker-type repos with registry url set to "registry.docker.io" (as the help message suggests) no longer works. Pulp returns a cryptic traceback where one is unable to tell the root cause of the failed sync: PLP0000: tuple index out of range traceback: | Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task R = retval = fun(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in __call__ return super(Task, self).__call__(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in __call__ return super(PulpTask, self).__call__(*args, **kwargs) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__ return self.run(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 796, in sync sync_report = sync_repo(transfer_repo, conduit, call_config) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 892, in wrap_f return f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 86, in sync_repo return self.sync_step.process_lifecycle() File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle super(PluginStep, self).process_lifecycle() File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle step.process() File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 257, in process self._process_block() File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 304, in _process_block self.process_main() File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py", line 235, in process_main self._process_manifest_list(manifest, digest, available_blobs, tag) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py", line 272, in _process_manifest_list tag=False) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/registry.py", line 388, in get_manifest response_headers, manifest = self._get_path(path, headers=request_headers) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/registry.py", line 497, in _get_path report = self.downloader.download_one(request) File "/usr/lib/python2.7/site-packages/nectar/downloaders/base.py", line 68, in download_one return self._download_one(request) File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 254, in _download_one return self._fetch(request, session) File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 378, in _fetch if nretry < DEFAULT_GENERIC_TRIES - 1 and e.args[1] == errno.ECONNRESET: IndexError: tuple index out of range Looks like the "e"xception args tuple doesn't contain more than one element. I guess the dockerhubs registry url changed. I tried to use `docker -D pull busybox` to see what url my docker client uses and found this: docker.io/library/busybox:latest however, changing the registry url to this makes katello automatically reach out to https://docker.io/v2/library/busybox/tags/list where it receives 404 Version-Release number of selected component (if applicable): 6.9.0-1 Steps to Reproduce: 1. try to create a custom repository with docker content-type, using the registry.docker.io as upstream url 2. use `busybox` image 3. try to sync Actual results: see above