+++ This bug was initially created as a clone of Bug #1515395 +++ Description of problem: Docker pull fails because issue with critical certificate extension ---------------- https://registry.redhat.io/containers/content/dist/rhel/server/7/7Server/x86_64/rhscl/1/containers/registry/rhscl/python-35-rhel7/manifests/sha256:be9df8f0385cb443c5c8ceabfa8b98aa3f213fa60ef1cd40c3649f650693df2e: tls: failed to parse certificate from server: x509: unhandled critical extension error: build error: unable to get registry.access.redhat.com/rhscl/python-35-rhel7@sha256:be9df8f0385cb443c5c8ceabfa8b98aa3f213fa60ef1cd40c3649f650693df2e ---------------- Environment is behind a proxy doing SSL decrypt/re-encrypt. As temporary workaround was implemented a 2nd proxy to decrypt-reencrypt the SSL again. In this way, docker do not see the 'bad' certificate. Version-Release number of selected component (if applicable): RHEL 7.4 docker-1.12.6-55 How reproducible: 100% Actual results: docker pull fails with tls: failed to parse certificate from server: x509: unhandled critical extension Expected results: docker pull image once the Additional info: The same problem reported https://github.com/moby/moby/issues/35152 and fix was committed https://go-review.googlesource.com/c/go/+/36900 --- Additional comment from Red Hat Bugzilla Rules Engine on 2017-11-20 13:06:14 EST --- Since this bug report was entered in Red Hat Bugzilla, the release flag has been set to ? to ensure that it is properly evaluated for this release. --- Additional comment from Antonio Murdaca on 2017-11-20 13:10:10 EST --- we need to wait till Golang in RHEL is 1.9.2 and we build docker with that golang unfortunately (we're probably at 1.8.x) --- Additional comment from Marc Jadoul on 2017-11-21 04:11:31 EST --- Hello, Please note it also concerns atomic-openshift-master-api: $ oc import-image ruby --all The import completed with errors. Name: ruby Namespace: openshift Created: 19 months ago Labels: <none> Annotations: openshift.io/display-name=Ruby openshift.io/image.dockerRepositoryCheck=2017-11-20T14:00:47Z Docker Pull Spec: 10.121.231.11:5000/openshift/ruby Image Lookup: local=false Unique Images: 7 Tags: 4 2.3 (latest) tagged from registry.access.redhat.com/rhscl/ruby-23-rhel7:latest Build and run Ruby 2.3 applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/blob/master/2.3/README.md'. Tags: builder, ruby Supports: ruby:2.3, ruby Example Repo: https://github.com/openshift/ruby-ex.git' ! error: Import failed (InternalError): Internal error occurred: Get https://registry.redhat.io/containers/content/dist/rhel/server/7/7Server/x86_64/rhscl/1/containers/registry/rhscl/ruby-23-rhel7/manifests/latest:' tls: failed to parse certificate from server: x509: unhandled critical extension 2 minutes ago * registry.access.redhat.com/rhscl/ruby-23-rhel7@sha256:a68e14c6be884e2d8f325850ff84e8e597c18756b177a54b2386dbafe48ab5f9' 10 days ago registry.access.redhat.com/rhscl/ruby-23-rhel7@sha256:236b125dc39ce8e307a12eb67f2d01100200c4d2e6e89f7b2e397b6fa7f9f81d' 5 weeks ago 2.2 tagged from registry.access.redhat.com/rhscl/ruby-22-rhel7:latest Build and run Ruby 2.2 applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/tree/master/2.2/README.md'. Tags: builder, ruby Supports: ruby:2.2, ruby Example Repo: https://github.com/openshift/ruby-ex.git' ! error: Import failed (InternalError): Internal error occurred: Get https://registry.redhat.io/containers/content/dist/rhel/server/7/7Server/x86_64/rhscl/1/containers/registry/rhscl/ruby-22-rhel7/manifests/latest:' tls: failed to parse certificate from server: x509: unhandled critical extension 2 minutes ago * registry.access.redhat.com/rhscl/ruby-22-rhel7@sha256:63068a00765c62a02fe69ab16da520ac2ff1458574f4c8c2c0cf300fbcfdc82e' 10 days ago registry.access.redhat.com/rhscl/ruby-22-rhel7@sha256:b72504509bd0db042594ac451974f362225e629e83f28f446088358f9f405463' 5 weeks ago 2.0 tagged from registry.access.redhat.com/openshift3/ruby-20-rhel7:latest Build and run Ruby 2.0 applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/tree/master/2.0/README.md'. Tags: hidden, builder, ruby Supports: ruby:2.0, ruby Example Repo: https://github.com/openshift/ruby-ex.git' ! error: Import failed (InternalError): Internal error occurred: Get https://registry.redhat.io/containers/content/dist/rhel/server/7/7Server/x86_64/ose/3/containers/registry/openshift3/ruby-20-rhel7/manifests/latest:' tls: failed to parse certificate from server: x509: unhandled critical extension 2 minutes ago * registry.access.redhat.com/openshift3/ruby-20-rhel7@sha256:9cfdf4b811ace13d4c555335b249ab831832a384113035512abc9d4d5cc59716' 9 months ago registry.access.redhat.com/openshift3/ruby-20-rhel7@sha256:9f8cfef74cefab63036ae16cac8766e76c0610a0c560fab83e093da740aa4369' 11 months ago registry.access.redhat.com/openshift3/ruby-20-rhel7:latest 19 months ago 15779e220dc9db16072d6f779bc816b16527e474958f089ac1d13cb7e5b5021c error: tag latest failed: Internal error occurred: Get https://registry.redhat.io/containers/content/dist/rhel/server/7/7Server/x86_64/ose/3/containers/registry/openshift3/ruby-20-rhel7/manifests/latest:' tls: failed to parse certificate from server: x509: unhandled critical extension error: tag latest failed: Internal error occurred: Get https://registry.redhat.io/containers/content/dist/rhel/server/7/7Server/x86_64/rhscl/1/containers/registry/rhscl/ruby-22-rhel7/manifests/latest:' tls: failed to parse certificate from server: x509: unhandled critical extension error: tag latest failed: Internal error occurred: Get https://registry.redhat.io/containers/content/dist/rhel/server/7/7Server/x86_64/rhscl/1/containers/registry/rhscl/ruby-23-rhel7/manifests/latest:' tls: failed to parse certificate from server: x509: unhandled critical extension
Sounds like this is blocked until openshift is built on top of go1.9.2 since the fix is in the go source. (same as https://bugzilla.redhat.com/show_bug.cgi?id=1515395#c2) Clayton, any idea when we're likely to start doing that?
ocp is being built on golang 1.9.2 now, so this should be fixed.
Honestly I don't think I can, maybe we can get some help from the originator, it sounds like a somewhat complicated setup w/ a proxy doing re-encryption. It might be sufficient to just point to registry.redhat.io but i'm not sure if it's still setup the way it was at the time this bug was filed. Jaroslav can you help our QE team understand how to validate that this is working now?
Hi Sorry, i was off a few days. I will jump on this asap, will update you about the status. Keeping need-info on me till finish reproducer Thanks !
Jaroslav Spanko, hi any progress for reproducing this bug? thanks
The simplest to implement reencrypting proxy is mitmproxy. I can probably create a key and cert having this issue. Then you should be able to test the correction?
I use the mitmproxy image to setup proxy-- $ docker run --rm -it -p 8080:8080 mitmproxy/mitmproxy, then launch an openshift cluster behind this proxy, add the created certificate 'mitmproxy-ca.pem' into /etc/pki/ca-trust/source/anchors/ on each instance. But still cannot reproduce it, does someone can give some suggestions
It will always fail... only with some certificates! To reproduce, you should have a certificate with the issue. I can try to create a certificate having the issue.... I think what cause the issue is a critical certificate extensions on the CA certificate, which was maybe not recognized by the go language.... My suspect: this extension on the CA... but I am not certain: nameconstraint: Autorisé=Aucun Exclus [1]Sous-arborescences (0..Max): Nom DNS=.ourdomain Tell me if I should try to create such certificate for you for testing.
Marc Jadoul, Please provide a certificate to reproduce this issue, thanks a lot
I tested and I can't reproduce the error either... the certificate did not apparently changed (but I do not have the old copy of it so I am not sure). The extension seems still there but not the issue. Or it has been solved already on my server.... even if it is still based on go 1.8.3..... $ docker version Client: Version: 1.12.6 API version: 1.24 Package version: docker-1.12.6-71.git3e8e77d.el7.x86_64 Go version: go1.8.3 I intended to only resign the bad certificate.... but I can't. Sorry....
Based on this: https://github.com/golang/go/issues/11091 I generated a new key and resigned the certificate. This certificate work with openssl, but not with docker.... But the error I get is NOT the same error. So I put the certificate here, but I supposes it is not the same issue.... I do not know what the initial issue with "unhandled critical extension" was and when/how it was fixed. I also do not know which fix is in go 1.9.2....
Created attachment 1403304 [details] mitmproxy-ca.pem with wrong Boolean encoding mitmproxy-ca.pem with wrong Boolean encoding which fail in GO but not in openssl. But this is probably an other issue as the error message is not the same. Replace this file in your .mitmproxy directory.
Marc Jadoul, thanks for your help. I have test with # docker version Client: Version: 1.12.6 API version: 1.24 Package version: docker-1.12.6-71.git3e8e77d.el7.x86_64 Go version: go1.8.3 Git commit: 3e8e77d/1.12.6 Built: Wed Dec 13 12:18:58 2017 OS/Arch: linux/amd64 could get error like below: Get https://registry.access.redhat.com/v1/_ping: tls: failed to parse certificate from server: asn1: syntax error: invalid boolean Refer to origin bug https://bugzilla.redhat.com/show_bug.cgi?id=1515395#c2 , I think it is related to docker if built of go 1.9 , not openshift. So change status to assigned. And maybe we also should change the target release to next version, not ocp 3.9
I took a look to initial links. The certificate I provided does trigger another unsolved go issue, which won't be solved. I will provide another certificate based on the one in https://go-review.googlesource.com/c/go/+/36900. Marc
Created attachment 1406760 [details] CA certificate for mithmproxy with critical name constraint extension with excluded domain. This bug should trigger the previously existing bug in GO which was fixed in go 1.9.2.
Verified Launch an openshift cluster with docker # docker version Version: 1.13.1 API version: 1.26 Package version: docker-1.13.1-58.git87f2fab.el7.x86_64 Go version: go1.9.2 Git commit: 87f2fab/1.13.1 Built: Mon Mar 19 18:55:01 2018 OS/Arch: linux/amd64 cannot reproduce this error, so move this bug to verified