Description of problem: After configure Docker registry to use OpenStack Swift Storage according to the documentation here https://docs.okd.io/3.11/install_config/registry/extended_registry_configuration.html. Docker registry Image get error Version-Release number of selected component (if applicable): Docker Registry Image: openshift/origin-docker-registry:v3.11.0 Actual results: panic: Swift authentication failed: Post https://auth.cloud.ovh.net/v2.0/tokens: x509: certificate is valid for *.cluster.octochroniq.de, cluster.octochroniq.de, not auth.cloud.ovh.net Additional info: Added certificate of auth.cloud.ovh.net as secret to service account "registry", nothing changed. Same configuration of the swift works when docker registry image started by docker commands. StackTrace: panic: Swift authentication failed: Post https://auth.cloud.ovh.net/v2.0/tokens: x509: certificate is valid for *.cluster.octochroniq.de, cluster.octochroniq.de, not auth.cloud.ovh.net goroutine 1 [running]: github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers.NewApp(0x7fc643824e60, 0x213f260, 0xc42007b180, 0x7fc643824e60) /tmp/openshift/build-rpms/rpm/BUILD/origin-dockerregistry-3.11.0/_output/local/go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers/app.go:124 +0x31fa github.com/openshift/image-registry/pkg/dockerregistry/server/supermiddleware.NewApp(0x17df020, 0x213f260, 0xc42007b180, 0x17e5ea0, 0xc4206db8c0, 0x17e7cc0) /tmp/openshift/build-rpms/rpm/BUILD/origin-dockerregistry-3.11.0/_output/local/go/src/github.com/openshift/image-registry/pkg/dockerregistry/server/supermiddleware/app.go:96 +0xae github.com/openshift/image-registry/pkg/dockerregistry/server.NewApp(0x17df020, 0x213f260, 0x17c7880, 0xc420170408, 0xc42007b180, 0xc4202c23c0, 0x0, 0x0, 0x5e, 0xc4207297f8) /tmp/openshift/build-rpms/rpm/BUILD/origin-dockerregistry-3.11.0/_output/local/go/src/github.com/openshift/image-registry/pkg/dockerregistry/server/app.go:137 +0x2ae github.com/openshift/image-registry/pkg/cmd/dockerregistry.NewServer(0x17df020, 0x213f260, 0xc42007b180, 0xc4202c23c0, 0x0, 0x0, 0x17f9c40) /tmp/openshift/build-rpms/rpm/BUILD/origin-dockerregistry-3.11.0/_output/local/go/src/github.com/openshift/image-registry/pkg/cmd/dockerregistry/dockerregistry.go:186 +0x19c github.com/openshift/image-registry/pkg/cmd/dockerregistry.Execute(0x17c4260, 0xc420170310) /tmp/openshift/build-rpms/rpm/BUILD/origin-dockerregistry-3.11.0/_output/local/go/src/github.com/openshift/image-registry/pkg/cmd/dockerregistry/dockerregistry.go:161 +0x9c3 main.main() /tmp/openshift/build-rpms/rpm/BUILD/origin-dockerregistry-3.11.0/_output/local/go/src/github.com/openshift/image-registry/cmd/dockerregistry/main.go:56 +0x403
Looks like the certificate provided by auth.cloud.ovh.net is not valid. Either contact the administrator of the auth server to update their certificate, or update the storage settings to set `insecureskipverify` setting to `true` [1]. [1] https://docs.docker.com/registry/storage-drivers/swift/
Certificates are valid, as you can find in the comments same configuration works when it is run as docker container only. It give error when it runs on OpenShift. "*.cluster.octochroniq.de" is the dns address of the openshift cluster it self. Why openshift tries to use selfsigned server certificates ?
Can you try `curl -v https://auth.cloud.ovh.net/v2.0/tokens` from within the registry container (oc -n default rsh dc/docker-registry)?
Hi, Below the result from command. Especially 159.69.113.247 is the server's ip not auth.cloud.ovh.net, why it tries to connect to server. I created a bash image directly on docker, ping resolved the true IP 51.68.65.247. bash-4.2$ curl -v https://auth.cloud.ovh.net/v2.0/tokens * About to connect() to auth.cloud.ovh.net port 443 (#0) * Trying 159.69.113.247... * Connected to auth.cloud.ovh.net (159.69.113.247) port 443 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * Server certificate: * subject: CN=*.cluster.octochroniq.de * start date: May 02 09:49:21 2019 GMT * expire date: May 01 09:49:22 2021 GMT * common name: *.cluster.octochroniq.de * issuer: CN=openshift-signer@1556790248 * NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER) * Peer's certificate issuer has been marked as not trusted by the user. * Closing connection 0
I'm not sure why this was assigned to my team.
Because it's a DNS problem: $ dig +noall +answer auth.cloud.ovh.net auth.cloud.ovh.net. 84 IN A 51.68.65.247 but from inside the pods it resolves to 159.69.113.247. $ whois 51.68.65.247 | grep ^org-name: org-name: OVH SAS $ whois 159.69.113.247 | grep ^org-name: | sort -u org-name: Hetzner Online GmbH
If you suspect a DNS issue, please provide: 1. Packet captures (tcpdump) of the request from inside the pod, and also from wherever the query working (the node?) 2. The /etc/resolv.conf from the pod, and from the node hosting the pod 3. The Dnsmasq configuration files from the node Thanks!
We are not using this feature anymore. We can close the issue. Thanks.