Bug 1518583

Summary: oc import-image pull fails due to extension certificate
Product: OpenShift Container Platform Reporter: Jaroslav Spanko <jspanko>
Component: ImageStreamsAssignee: Ben Parees <bparees>
Status: CLOSED CURRENTRELEASE QA Contact: Dongbo Yan <dyan>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.6.0CC: amurdaca, aos-bugs, atomic-bugs, bparees, ccoleman, dornelas, dwalsh, haowang, jokerman, jspanko, lsm5, marc.jadoul, mmccomas, rhowe, vlaad, wzheng
Target Milestone: ---Keywords: Extras
Target Release: 3.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Additional certificate name constraints prevented valid certificates from being processed, resulting in error "tls: failed to parse certificate from server: x509: unhandled critical extension" Consequence: Valid certificates were unusable. Fix: Moved to newer golang libraries that fixed the constraint. Result: Certificates that previously failed can now be used.
Story Points: ---
Clone Of: 1515395 Environment:
Last Closed: 2018-08-29 13:55:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1515395    
Bug Blocks:    
Attachments:
Description Flags
mitmproxy-ca.pem with wrong Boolean encoding
none
CA certificate for mithmproxy with critical name constraint extension with excluded domain. none

Description Jaroslav Spanko 2017-11-29 09:09:59 UTC
+++ This bug was initially created as a clone of Bug #1515395 +++

Description of problem:
Docker pull fails because issue with critical certificate extension
----------------
https://registry.redhat.io/containers/content/dist/rhel/server/7/7Server/x86_64/rhscl/1/containers/registry/rhscl/python-35-rhel7/manifests/sha256:be9df8f0385cb443c5c8ceabfa8b98aa3f213fa60ef1cd40c3649f650693df2e: tls: failed to parse certificate from server: x509: unhandled critical extension error: build error: unable to get registry.access.redhat.com/rhscl/python-35-rhel7@sha256:be9df8f0385cb443c5c8ceabfa8b98aa3f213fa60ef1cd40c3649f650693df2e
----------------

Environment is behind a proxy doing SSL decrypt/re-encrypt. As temporary workaround was implemented
a 2nd proxy to decrypt-reencrypt the SSL again. In this way, docker do not see the 'bad' certificate.

Version-Release number of selected component (if applicable):
RHEL 7.4
docker-1.12.6-55

How reproducible:
100%

Actual results:
docker pull fails with tls: failed to parse certificate from server: x509: unhandled critical extension

Expected results:
docker pull image once the 

Additional info:
The same problem reported 
https://github.com/moby/moby/issues/35152

and fix was committed  
https://go-review.googlesource.com/c/go/+/36900

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-11-20 13:06:14 EST ---

Since this bug report was entered in Red Hat Bugzilla, the release flag has been set to ? to ensure that it is properly evaluated for this release.

--- Additional comment from Antonio Murdaca on 2017-11-20 13:10:10 EST ---

we need to wait till Golang in RHEL is 1.9.2 and we build docker with that golang unfortunately (we're probably at 1.8.x)

--- Additional comment from Marc Jadoul on 2017-11-21 04:11:31 EST ---

Hello,
Please note it also concerns atomic-openshift-master-api:

$ oc import-image  ruby --all
The import completed with errors.

Name:                   ruby
Namespace:              openshift
Created:                19 months ago
Labels:                 <none>
Annotations:            openshift.io/display-name=Ruby
                        openshift.io/image.dockerRepositoryCheck=2017-11-20T14:00:47Z
Docker Pull Spec:       10.121.231.11:5000/openshift/ruby
Image Lookup:           local=false
Unique Images:          7
Tags:                   4

2.3 (latest)
  tagged from registry.access.redhat.com/rhscl/ruby-23-rhel7:latest

  Build and run Ruby 2.3 applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/blob/master/2.3/README.md'.
  Tags: builder, ruby
  Supports: ruby:2.3, ruby
  Example Repo: https://github.com/openshift/ruby-ex.git'

  ! error: Import failed (InternalError): Internal error occurred: Get https://registry.redhat.io/containers/content/dist/rhel/server/7/7Server/x86_64/rhscl/1/containers/registry/rhscl/ruby-23-rhel7/manifests/latest:' tls: failed to parse certificate from server: x509: unhandled critical extension
      2 minutes ago
  * registry.access.redhat.com/rhscl/ruby-23-rhel7@sha256:a68e14c6be884e2d8f325850ff84e8e597c18756b177a54b2386dbafe48ab5f9'
      10 days ago
    registry.access.redhat.com/rhscl/ruby-23-rhel7@sha256:236b125dc39ce8e307a12eb67f2d01100200c4d2e6e89f7b2e397b6fa7f9f81d'
      5 weeks ago

2.2
  tagged from registry.access.redhat.com/rhscl/ruby-22-rhel7:latest

  Build and run Ruby 2.2 applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/tree/master/2.2/README.md'.
  Tags: builder, ruby
  Supports: ruby:2.2, ruby
  Example Repo: https://github.com/openshift/ruby-ex.git'

  ! error: Import failed (InternalError): Internal error occurred: Get https://registry.redhat.io/containers/content/dist/rhel/server/7/7Server/x86_64/rhscl/1/containers/registry/rhscl/ruby-22-rhel7/manifests/latest:' tls: failed to parse certificate from server: x509: unhandled critical extension
      2 minutes ago
  * registry.access.redhat.com/rhscl/ruby-22-rhel7@sha256:63068a00765c62a02fe69ab16da520ac2ff1458574f4c8c2c0cf300fbcfdc82e'
      10 days ago
    registry.access.redhat.com/rhscl/ruby-22-rhel7@sha256:b72504509bd0db042594ac451974f362225e629e83f28f446088358f9f405463'
      5 weeks ago

2.0
  tagged from registry.access.redhat.com/openshift3/ruby-20-rhel7:latest

  Build and run Ruby 2.0 applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/tree/master/2.0/README.md'.
  Tags: hidden, builder, ruby
  Supports: ruby:2.0, ruby
  Example Repo: https://github.com/openshift/ruby-ex.git'

  ! error: Import failed (InternalError): Internal error occurred: Get https://registry.redhat.io/containers/content/dist/rhel/server/7/7Server/x86_64/ose/3/containers/registry/openshift3/ruby-20-rhel7/manifests/latest:' tls: failed to parse certificate from server: x509: unhandled critical extension
      2 minutes ago
  * registry.access.redhat.com/openshift3/ruby-20-rhel7@sha256:9cfdf4b811ace13d4c555335b249ab831832a384113035512abc9d4d5cc59716'
      9 months ago
    registry.access.redhat.com/openshift3/ruby-20-rhel7@sha256:9f8cfef74cefab63036ae16cac8766e76c0610a0c560fab83e093da740aa4369'
      11 months ago
    registry.access.redhat.com/openshift3/ruby-20-rhel7:latest
      19 months ago     15779e220dc9db16072d6f779bc816b16527e474958f089ac1d13cb7e5b5021c

error: tag latest failed: Internal error occurred: Get https://registry.redhat.io/containers/content/dist/rhel/server/7/7Server/x86_64/ose/3/containers/registry/openshift3/ruby-20-rhel7/manifests/latest:' tls: failed to parse certificate from server: x509: unhandled critical extension
error: tag latest failed: Internal error occurred: Get https://registry.redhat.io/containers/content/dist/rhel/server/7/7Server/x86_64/rhscl/1/containers/registry/rhscl/ruby-22-rhel7/manifests/latest:' tls: failed to parse certificate from server: x509: unhandled critical extension
error: tag latest failed: Internal error occurred: Get https://registry.redhat.io/containers/content/dist/rhel/server/7/7Server/x86_64/rhscl/1/containers/registry/rhscl/ruby-23-rhel7/manifests/latest:' tls: failed to parse certificate from server: x509: unhandled critical extension

Comment 2 Ben Parees 2017-12-04 17:50:34 UTC
Sounds like this is blocked until openshift is built on top of go1.9.2 since the fix is in the go source. (same as https://bugzilla.redhat.com/show_bug.cgi?id=1515395#c2)


Clayton, any idea when we're likely to start doing that?

Comment 3 Ben Parees 2018-01-18 16:23:55 UTC
ocp is being built on golang 1.9.2 now, so this should be fixed.

Comment 5 Ben Parees 2018-01-26 15:25:11 UTC
Honestly I don't think I can, maybe we can get some help from the originator, it sounds like a somewhat complicated setup w/ a proxy doing re-encryption.

It might be sufficient to just point to registry.redhat.io but i'm not sure if it's still setup the way it was at the time this bug was filed.

Jaroslav can you help our QE team understand how to validate that this is working now?

Comment 6 Jaroslav Spanko 2018-02-05 11:43:34 UTC
Hi
Sorry, i was off a few days.
I will jump on this asap, will update you about the status.
Keeping need-info on me till finish reproducer 
Thanks !

Comment 7 Dongbo Yan 2018-02-22 02:59:38 UTC
Jaroslav Spanko, hi
any progress for reproducing this bug? thanks

Comment 8 Marc Jadoul 2018-02-22 19:04:44 UTC
The simplest to implement reencrypting proxy is mitmproxy. I can probably create a key and cert having this issue. Then you should be able to test the correction?

Comment 9 Dongbo Yan 2018-03-01 11:34:14 UTC
I use the mitmproxy image to setup proxy--
$ docker run --rm -it -p 8080:8080 mitmproxy/mitmproxy,
then launch an openshift cluster behind this proxy, add the created certificate 'mitmproxy-ca.pem' into /etc/pki/ca-trust/source/anchors/ on each instance.
But still cannot reproduce it, does someone can give some suggestions

Comment 10 Marc Jadoul 2018-03-01 17:29:08 UTC
It will always fail... only with some certificates!
To reproduce, you should have a certificate with the issue.
I can try to create a certificate having the issue....

I think what cause the issue is a critical certificate extensions on the CA certificate, which was maybe not recognized by the go language....

My suspect: this extension on the CA... but I am not certain:
nameconstraint:
Autorisé=Aucun
Exclus
     [1]Sous-arborescences (0..Max):
          Nom DNS=.ourdomain

Tell me if I should try to create such certificate for you for testing.

Comment 11 Dongbo Yan 2018-03-02 02:10:27 UTC
Marc Jadoul,
Please provide a certificate to reproduce this issue, thanks a lot

Comment 12 Marc Jadoul 2018-03-02 18:49:41 UTC
I tested and I can't reproduce the error either... the certificate did not apparently changed (but I do not have the old copy of it so I am not sure).

The extension seems still there but not the issue.

Or it has been solved already on my server.... even if it is still based on go 1.8.3.....

$ docker version
Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-71.git3e8e77d.el7.x86_64
 Go version:      go1.8.3

I intended to only resign the bad certificate.... but I can't.
Sorry....

Comment 13 Marc Jadoul 2018-03-03 08:22:26 UTC
Based on this: https://github.com/golang/go/issues/11091
I generated a new key and resigned the certificate.
This certificate work with openssl, but not with docker....
But the error I get is NOT the same error.
So I put the certificate here, but I supposes it is not the same issue....

I do not know what the initial issue with "unhandled critical extension" was and when/how it was fixed. I also do not know which fix is in go 1.9.2....

Comment 14 Marc Jadoul 2018-03-03 08:26:16 UTC
Created attachment 1403304 [details]
mitmproxy-ca.pem with wrong Boolean encoding

mitmproxy-ca.pem with wrong Boolean encoding which fail in GO but not in openssl.
But this is probably an other issue as the error message is not the same.
Replace this file in your .mitmproxy directory.

Comment 15 Dongbo Yan 2018-03-06 07:21:08 UTC
Marc Jadoul, thanks for your help.
I have test with # docker version
Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-71.git3e8e77d.el7.x86_64
 Go version:      go1.8.3
 Git commit:      3e8e77d/1.12.6
 Built:           Wed Dec 13 12:18:58 2017
 OS/Arch:         linux/amd64

could get error like below:
Get https://registry.access.redhat.com/v1/_ping: tls: failed to parse certificate from server: asn1: syntax error: invalid boolean

Refer to origin bug  https://bugzilla.redhat.com/show_bug.cgi?id=1515395#c2 , I think it is related to docker if built of go 1.9 , not openshift. So change status to assigned.

And maybe we also should change the target release to next version, not ocp 3.9

Comment 16 Marc Jadoul 2018-03-06 08:41:54 UTC
I took a look to initial links.
The certificate I provided does trigger another unsolved go issue, which won't be solved. I will provide another certificate based on the one in https://go-review.googlesource.com/c/go/+/36900.

Marc

Comment 17 Marc Jadoul 2018-03-11 08:50:55 UTC
Created attachment 1406760 [details]
CA certificate for mithmproxy with critical name constraint extension with excluded domain.

This bug should trigger the previously existing bug in GO which was fixed in go 1.9.2.

Comment 18 Dongbo Yan 2018-05-07 10:14:10 UTC
Verified
Launch an openshift cluster with docker
# docker version

 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-58.git87f2fab.el7.x86_64
 Go version:      go1.9.2
 Git commit:      87f2fab/1.13.1
 Built:           Mon Mar 19 18:55:01 2018
 OS/Arch:         linux/amd64

cannot reproduce this error, so move this bug to verified