Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1706030

Summary: Docker Image Registry - Swift Configuration - Certificate Error
Product: OpenShift Container Platform Reporter: Aytunc BEKEN <aytuncbeken.ab>
Component: NetworkingAssignee: Dan Mace <dmace>
Networking sub component: DNS QA Contact: Hongan Li <hongli>
Status: CLOSED WONTFIX Docs Contact:
Severity: medium    
Priority: unspecified CC: adam.kaplan, aos-bugs
Version: 3.11.0Keywords: Reopened
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-16 13:39:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Aytunc BEKEN 2019-05-03 12:13:17 UTC
Description of problem: After configure Docker registry to use OpenStack Swift Storage according to the documentation here https://docs.okd.io/3.11/install_config/registry/extended_registry_configuration.html. Docker registry Image get error


Version-Release number of selected component (if applicable):
Docker Registry Image: openshift/origin-docker-registry:v3.11.0

Actual results:
panic: Swift authentication failed: Post https://auth.cloud.ovh.net/v2.0/tokens: x509: certificate is valid for *.cluster.octochroniq.de, cluster.octochroniq.de, not auth.cloud.ovh.net

Additional info:
Added certificate of auth.cloud.ovh.net as secret to service account "registry", nothing changed. Same configuration of the swift works when docker registry image started by docker commands.

StackTrace:
panic: Swift authentication failed: Post https://auth.cloud.ovh.net/v2.0/tokens:  x509: certificate is valid for *.cluster.octochroniq.de, cluster.octochroniq.de, not auth.cloud.ovh.net

goroutine 1 [running]:
github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers.NewApp(0x7fc643824e60, 0x213f260, 0xc42007b180, 0x7fc643824e60)
	/tmp/openshift/build-rpms/rpm/BUILD/origin-dockerregistry-3.11.0/_output/local/go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers/app.go:124 +0x31fa
github.com/openshift/image-registry/pkg/dockerregistry/server/supermiddleware.NewApp(0x17df020, 0x213f260, 0xc42007b180, 0x17e5ea0, 0xc4206db8c0, 0x17e7cc0)
	/tmp/openshift/build-rpms/rpm/BUILD/origin-dockerregistry-3.11.0/_output/local/go/src/github.com/openshift/image-registry/pkg/dockerregistry/server/supermiddleware/app.go:96 +0xae
github.com/openshift/image-registry/pkg/dockerregistry/server.NewApp(0x17df020, 0x213f260, 0x17c7880, 0xc420170408, 0xc42007b180, 0xc4202c23c0, 0x0, 0x0, 0x5e, 0xc4207297f8)
	/tmp/openshift/build-rpms/rpm/BUILD/origin-dockerregistry-3.11.0/_output/local/go/src/github.com/openshift/image-registry/pkg/dockerregistry/server/app.go:137 +0x2ae
github.com/openshift/image-registry/pkg/cmd/dockerregistry.NewServer(0x17df020, 0x213f260, 0xc42007b180, 0xc4202c23c0, 0x0, 0x0, 0x17f9c40)
	/tmp/openshift/build-rpms/rpm/BUILD/origin-dockerregistry-3.11.0/_output/local/go/src/github.com/openshift/image-registry/pkg/cmd/dockerregistry/dockerregistry.go:186 +0x19c
github.com/openshift/image-registry/pkg/cmd/dockerregistry.Execute(0x17c4260, 0xc420170310)
	/tmp/openshift/build-rpms/rpm/BUILD/origin-dockerregistry-3.11.0/_output/local/go/src/github.com/openshift/image-registry/pkg/cmd/dockerregistry/dockerregistry.go:161 +0x9c3
main.main()
/tmp/openshift/build-rpms/rpm/BUILD/origin-dockerregistry-3.11.0/_output/local/go/src/github.com/openshift/image-registry/cmd/dockerregistry/main.go:56 +0x403

Comment 1 Adam Kaplan 2019-05-03 12:50:33 UTC
Looks like the certificate provided by auth.cloud.ovh.net is not valid. Either contact the administrator of the auth server to update their certificate, or update the storage settings to set `insecureskipverify` setting to `true` [1].

[1] https://docs.docker.com/registry/storage-drivers/swift/

Comment 2 Aytunc BEKEN 2019-05-03 12:57:15 UTC
Certificates are valid, as you can find in the comments same configuration works when it is run as docker container only. It give error when it runs on OpenShift.
"*.cluster.octochroniq.de" is the dns address of the openshift cluster it self. Why openshift tries to use selfsigned server certificates ?

Comment 3 Oleg Bulatov 2019-05-04 13:03:24 UTC
Can you try `curl -v https://auth.cloud.ovh.net/v2.0/tokens` from within the registry container (oc -n default rsh dc/docker-registry)?

Comment 4 Aytunc BEKEN 2019-05-06 07:48:04 UTC
Hi,
Below the result from command. Especially 159.69.113.247 is the server's ip not auth.cloud.ovh.net, why it tries to connect to server. 

I created a bash image directly on docker, ping resolved the true IP 51.68.65.247.

bash-4.2$ curl -v https://auth.cloud.ovh.net/v2.0/tokens
* About to connect() to auth.cloud.ovh.net port 443 (#0)
*   Trying 159.69.113.247...
* Connected to auth.cloud.ovh.net (159.69.113.247) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* Server certificate:
* 	subject: CN=*.cluster.octochroniq.de
* 	start date: May 02 09:49:21 2019 GMT
* 	expire date: May 01 09:49:22 2021 GMT
* 	common name: *.cluster.octochroniq.de
* 	issuer: CN=openshift-signer@1556790248
* NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER)
* Peer's certificate issuer has been marked as not trusted by the user.
* Closing connection 0

Comment 5 Dan Mace 2019-10-11 01:37:14 UTC
I'm not sure why this was assigned to my team.

Comment 6 Oleg Bulatov 2019-10-14 13:15:47 UTC
Because it's a DNS problem:

$ dig +noall +answer auth.cloud.ovh.net
auth.cloud.ovh.net.	84	IN	A	51.68.65.247

but from inside the pods it resolves to 159.69.113.247.

$ whois 51.68.65.247 | grep ^org-name:
org-name:       OVH SAS
$ whois 159.69.113.247 | grep ^org-name: | sort -u
org-name:       Hetzner Online GmbH

Comment 7 Dan Mace 2019-10-14 14:16:17 UTC
If you suspect a DNS issue, please provide:

1. Packet captures (tcpdump) of the request from inside the pod, and also from wherever the query working (the node?)
2. The /etc/resolv.conf from the pod, and from the node hosting the pod
3. The Dnsmasq configuration files from the node

Thanks!

Comment 8 Aytunc BEKEN 2019-10-16 13:39:45 UTC
We are not using this feature anymore. We can close the issue. 

Thanks.