Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1433458

Summary: DKR1008: Could not find registry API at https://docker-registry.engineering.redhat.com
Product: Red Hat Satellite Reporter: Tom McKay <tomckay>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: jcallaha
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.6CC: adahms, arahaman, bbuckingham, bkearney, bmbouter, daviddavis, dkliban, ehelms, ggainey, ipanova, jcallaha, lucarval, mhrivnak, oshtaier, paji, pcreech, rchan, satellite6-bugs, ttereshc
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pulp-2.13.0 Doc Type: Release Note
Doc Text:
To synchronize container images from a registry with self-signed certificates, you must either configure certificates manually or disable the SSL Verify option.
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
failed sync none

Description Tom McKay 2017-03-17 16:47:19 UTC
https://pulp.plan.io/issues/2643

Internal registry https://docker-registry.engineering.redhat.com cannot sync repos into foreman. 'docker pull' works

docker pull docker-registry.engineering.redhat.com/thomasmckay/freewill:v2
Packages:

[root@devel ~]# rpm -qa | grep pulp
pulp-server-2.12.0-1.el7.noarch
pulp-puppet-plugins-2.12.0-1.el7.noarch
pulp-selinux-2.12.0-1.el7.noarch
python-pulp-client-lib-2.12.0-1.el7.noarch
python-pulp-ostree-common-1.2.0-1.el7.noarch
pulp-client-1.0-1.noarch
python-pulp-docker-common-2.3.0-1.el7.noarch
pulp-docker-admin-extensions-2.3.0-1.el7.noarch
python-pulp-puppet-common-2.12.0-1.el7.noarch
python-pulp-oid_validation-2.12.0-1.el7.noarch
pulp-rpm-admin-extensions-2.12.0-1.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-6.pulp.el7.noarch
python-pulp-common-2.12.0-1.el7.noarch
python-pulp-rpm-common-2.12.0-1.el7.noarch
pulp-admin-client-2.12.0-1.el7.noarch
pulp-puppet-tools-2.12.0-1.el7.noarch
pulp-docker-plugins-2.3.0-1.el7.noarch
python-pulp-bindings-2.12.0-1.el7.noarch
pulp-ostree-plugins-1.2.0-1.el7.noarch
pulp-katello-1.0.2-1.el7.noarch
python-pulp-repoauth-2.12.0-1.el7.noarch
pulp-rpm-plugins-2.12.0-1.el7.noarch
python-pulp-streamer-2.12.0-1.el7.noarch
rubygem-smart_proxy_pulp-1.3.0-1.el7.noarch
Log:

Mar 17 15:36:30 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: Running task : [9337920c-83ba-454f-a9b4-8cfb79d8f478]
Mar 17 15:36:30 devel.example.com pulp[10325]: pulp_docker.plugins.importers.sync:DEBUG: v1 API skipped due to config
Mar 17 15:36:30 devel.example.com pulp[10325]: pulp_docker.plugins.registry:DEBUG: Determining if the registry URL can do v2 of the Docker API.
Mar 17 15:36:30 devel.example.com pulp[10325]: pulp_docker.plugins.registry:DEBUG: Retrieving https://docker-registry.engineering.redhat.com/v2/
Mar 17 15:36:30 devel.example.com pulp[10325]: nectar.downloaders.threaded:DEBUG: Attempting to connect to https://docker-registry.engineering.redhat.com/v2/.
Mar 17 15:36:30 devel.example.com pulp[10325]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): docker-registry.engineering.redhat.com
Mar 17 15:36:30 devel.example.com pulp[10312]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[4d1f4ce3-c31c-44c6-8637-301bbb902a2a] succeeded in 0.0419807829894s: None
Mar 17 15:36:30 devel.example.com pulp[10180]: celery.worker.job:DEBUG: Task accepted: pulp.server.managers.repo.sync.sync[9337920c-83ba-454f-a9b4-8cfb79d8f478] pid:10325
Mar 17 15:36:30 devel.example.com pulp[10180]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[2bff1f0d-b39c-435a-97b9-dff17929a209]
Mar 17 15:36:30 devel.example.com pulp[10456]: pulp.server.webservices.views.decorators:DEBUG: User preauthenticated: admin
Mar 17 15:36:31 devel.example.com pulp[10325]: requests.packages.urllib3.connectionpool:DEBUG: "GET /v2/ HTTP/1.1" 401 87
Mar 17 15:36:31 devel.example.com pulp[10325]: nectar.downloaders.threaded:DEBUG: download failed: Download of https://docker-registry.engineering.redhat.com/v2/ failed with code 401: Unauthorized
Mar 17 15:36:31 devel.example.com pulp[10325]: pulp_docker.plugins.registry:DEBUG: Download unauthorized, attempting to retrieve a token.
Mar 17 15:36:31 devel.example.com pulp[10325]: pulp_docker.plugins.token_util:DEBUG: Requesting token from https://docker-registry.engineering.redhat.com/openshift/token
Mar 17 15:36:31 devel.example.com pulp[10325]: nectar.downloaders.threaded:DEBUG: Attempting to connect to https://docker-registry.engineering.redhat.com/openshift/token.
Mar 17 15:36:31 devel.example.com pulp[10325]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): docker-registry.engineering.redhat.com
Mar 17 15:36:31 devel.example.com pulp[10456]: pulp.server.webservices.views.decorators:DEBUG: User preauthenticated: admin
Mar 17 15:36:31 devel.example.com pulp[10325]: requests.packages.urllib3.connectionpool:DEBUG: "GET /openshift/token HTTP/1.1" 200 49
Mar 17 15:36:31 devel.example.com pulp[10325]: nectar.downloaders.threaded:DEBUG: Attempting to connect to https://docker-registry.engineering.redhat.com/v2/.
Mar 17 15:36:31 devel.example.com pulp[10325]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): docker-registry.engineering.redhat.com
Mar 17 15:36:31 devel.example.com pulp[10456]: pulp.server.webservices.views.decorators:DEBUG: User preauthenticated: admin
Mar 17 15:36:32 devel.example.com pulp[10325]: requests.packages.urllib3.connectionpool:DEBUG: "GET /v2/ HTTP/1.1" 401 87
Mar 17 15:36:32 devel.example.com pulp[10325]: nectar.downloaders.threaded:DEBUG: download failed: Download of https://docker-registry.engineering.redhat.com/v2/ failed with code 401: Unauthorized
Mar 17 15:36:32 devel.example.com pulp[10459]: pulp.server.webservices.views.decorators:DEBUG: User preauthenticated: admin
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:INFO: Task failed : [9337920c-83ba-454f-a9b4-8cfb79d8f478] : Could not find registry API at https://docker-registry.engineering.redhat.com
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296) Traceback (most recent call last):
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)     R = retval = fun(*args, **kwargs)
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 505, in __call__
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)     return super(Task, self).__call__(*args, **kwargs)
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 108, in __call__
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)     return super(PulpTask, self).__call__(*args, **kwargs)
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)     return self.run(*args, **kwargs)
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 762, in sync
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)     sync_report = sync_repo(transfer_repo, conduit, call_config)
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 708, in wrap_f
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)     return f(*args, **kwargs)
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)   File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 83, in sync_repo
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)     self.sync_step = sync.SyncStep(repo=repo, conduit=sync_conduit, config=config)
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)   File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py", line 89, in __init__
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)     raise PulpCodedException(error_code=error_codes.DKR1008, registry=url)
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296) PulpCodedException: Could not find registry API at https://docker-registry.engineering.redhat.com
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)

Comment 3 Tom McKay 2017-03-18 02:22:34 UTC
Looks identical except for the dates, could that be an issue?


$ curl -k -I https://docker-registry.engineering.redhat.com/v2
HTTP/2.0 301
docker-distribution-api-version:registry/2.0
location:/v2/
content-type:text/plain; charset=utf-8
date:Fri, 17 Mar 2017 22:38:30 GMT

$ curl -k -I https://docker-registry.engineering.redhat.com/v2/
HTTP/2.0 401
content-type:application/json; charset=utf-8
docker-distribution-api-version:registry/2.0
www-authenticate:Bearer realm="https://docker-registry.engineering.redhat.com/openshift/token"
content-length:87
date:Fri, 17 Mar 2017 22:38:38 GMT



$ curl -k -I https://atomic-registry.usersys.redhat.com:5000/v2
HTTP/2.0 301
docker-distribution-api-version:registry/2.0
location:/v2/
content-type:text/plain; charset=utf-8
date:Sat, 18 Mar 2017 02:17:41 GMT

$ curl -k -I https://atomic-registry.usersys.redhat.com:5000/v2/
HTTP/2.0 401
content-type:application/json; charset=utf-8
docker-distribution-api-version:registry/2.0
www-authenticate:Bearer realm="https://atomic-registry.usersys.redhat.com:5000/openshift/token"
x-registry-supports-signatures:1
content-length:87
date:Sat, 18 Mar 2017 02:17:47 GMT

Comment 4 Tom McKay 2017-03-18 02:31:04 UTC
From what I understand, the atomic-registry is using GoogleAuthProvider while docker-registry is using LDAPPasswordIdentityProvider.

Comment 5 Luiz Carvalho 2017-03-20 15:17:35 UTC
I don't see a discrepancy with the dates:

$ curl -k -I https://atomic-registry.usersys.redhat.com:5000/v2/
HTTP/2 401 
content-type: application/json; charset=utf-8
docker-distribution-api-version: registry/2.0
www-authenticate: Bearer realm="https://atomic-registry.usersys.redhat.com:5000/openshift/token"
x-registry-supports-signatures: 1
content-length: 87
date: Mon, 20 Mar 2017 15:14:05 GMT

$ curl -k -I https://docker-registry.engineering.redhat.com/v2/
HTTP/2 401 
content-type: application/json; charset=utf-8
docker-distribution-api-version: registry/2.0
www-authenticate: Bearer realm="https://docker-registry.engineering.redhat.com/openshift/token"
content-length: 87
date: Mon, 20 Mar 2017 15:14:19 GMT

Although I do see that atomic-registry returns the addtional header: 
x-registry-supports-signatures:1

Comment 6 Tom McKay 2017-03-20 17:46:07 UTC
From @ipanova

--- failure ---

[ipanova@ipanova ~]$ curl -k -I -L -H 'Authorization: Bearer anonymous' https://atomic-registry.usersys.redhat.com:5000/v2/
HTTP/1.1 200 OK
Content-Length: 2
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
X-Registry-Supports-Signatures: 1
Date: Mon, 20 Mar 2017 17:30:52 GMT

[ipanova@ipanova ~]$ curl -k -I -L -H 'Authorization: Bearer anonymous' https://docker-registry.engineering.redhat.com/v2
HTTP/1.1 301 Moved Permanently
Docker-Distribution-Api-Version: registry/2.0
Location: /v2/
Date: Mon, 20 Mar 2017 17:31:18 GMT
Content-Type: text/plain; charset=utf-8

HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Basic realm=openshift,error="access denied"
Date: Mon, 20 Mar 2017 17:31:18 GMT
Content-Length: 87

--- failure ---

$ curl -k -I -L -H 'Authorization: Bearer ...snipped token...' https://registry-1.docker.io/v2
HTTP/1.1 301 Moved Permanently
Docker-Distribution-Api-Version: registry/2.0
Location: /v2/
Date: Mon, 20 Mar 2017 17:34:24 GMT
Content-Type: text/plain; charset=utf-8
Strict-Transport-Security: max-age=31536000

HTTP/1.1 200 OK
Content-Length: 2
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Date: Mon, 20 Mar 2017 17:34:24 GMT
Strict-Transport-Security: max-age=31536000

Comment 7 pulp-infra@redhat.com 2017-03-21 19:17:51 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 8 pulp-infra@redhat.com 2017-03-21 19:17:56 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 9 Ina Panova 2017-03-22 14:27:37 UTC
more info about this issue https://pulp.plan.io/issues/2643#note-4 and https://pulp.plan.io/issues/2643#note-5

Comment 10 pulp-infra@redhat.com 2017-03-23 13:32:08 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 11 Brian Bouterse 2017-04-11 18:05:25 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 12 pulp-infra@redhat.com 2017-04-11 18:20:22 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 13 pulp-infra@redhat.com 2017-04-17 13:32:54 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 14 pulp-infra@redhat.com 2017-04-27 14:03:47 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 16 Ashfaqur Rahaman 2017-08-08 01:19:27 UTC
Hello,

Is there any work around in pulp-docker-plugins-2.0.3-1 version ? Satellite version : 6.2.8 

We are facing issue while syncing docker repo : https://docker.io 

************************
Error:

Katello::Errors::PulpError

DKR1008: Could not find registry API at https://docker.io
************************

Thanks
Ash

Comment 21 jcallaha 2017-11-02 19:47:16 UTC
Failed QA in Satellite 6.3 Beta Snap 22.

Still seeing the issue persisting as initially reported. See attached screenshot and below pulp-admin session.


pulp docker repo create --feed=https://docker-registry.engineering.redhat.com --upstream-name=thomasmckay/freewill --repo-id=eric
Repository [eric] successfully created

[vagrant@obelix ~]$ pulp docker repo sync run --repo-id=eric
+----------------------------------------------------------------------+
                    Synchronizing Repository [eric]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.



Task Failed

Could not find registry API at https://docker-registry.engineering.redhat.com

Comment 22 jcallaha 2017-11-02 19:47:43 UTC
Created attachment 1347110 [details]
failed sync

Comment 23 pulp-infra@redhat.com 2017-11-02 20:03:00 UTC
Requesting needsinfo from upstream developer ipanova because the 'FailedQA' flag is set.

Comment 26 jcallaha 2017-11-03 15:07:40 UTC
Verified in Satellite 6.3 Snap 22.

Re-tested after changing the SSL Verify field to No. The repo was successfully synced without any of the previously seen issues. See attached screenshot for verification.

Comment 27 Bryan Kearney 2018-02-21 17:34:24 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0336

Comment 28 Bryan Kearney 2018-02-21 17:34:26 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0336