Description of problem: When using a custom S3 storage provider (e.g. minio), I am not able to configure a CA certificate for my docker-registry to use when calling to the https endpoint. This results in x509 errors in docker-registry and openshift app build failures. 172.56.8.1 - - [14/Jun/2018:14:41:11 +0000] "POST /v2/mytest/ruby-ex/blobs/uploads/ HTTP/1.1" 500 104 "" "docker/1.13.1 go/go1.9.2 kernel/3.10.0-862.3.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(go-dockerclient)" time="2018-06-14T14:41:11.813874015Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: RequestError: send request failed caused by: Put https://s3.example.com/registry/registry/docker/registry/v2/repositories/mytest/ruby-ex/_uploads/9fd2ad5b-2639-4557-b3a0-ce34d4ff75dc/startedat: x509: certificate signed by unknown authority" err.message="unknown error" go.version=go1.9.4 http.request.host="docker-registry.default.svc:5000" http.request.id=f9e00c45-d493-45ef-bbc4-ac596726a797 http.request.method=POST http.request.remoteaddr="172.56.8.1:39522" http.request.uri=/v2/mytest/ruby-ex/blobs/uploads/ http.request.useragent="docker/1.13.1 go/go1.9.2 kernel/3.10.0-862.3.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(go-dockerclient)" http.response.contenttype="application/json; charset=utf-8" http.response.duration=459.224805ms http.response.status=500 http.response.written=104 instance.id=8b028ba4-37cd-4548-bce6-95e934565039 openshift.auth.user="system:serviceaccount:mytest:builder" vars.name=mytest/ruby-ex Version-Release number of selected component (if applicable): 3.7 and 3.9 How reproducible: Always Steps to Reproduce: 1. Deploy standalone registry with s3 backed storage and https endpoint with self-signed or otherwise untrusted certificate. See Option D: https://docs.openshift.com/container-platform/3.7/install_config/install/advanced_install.html#advanced-install-registry-storage 2. oc new-project mytest 3. oc new-app .... 4. Watch build pod and docker-registry logs for failure Expected results: I would expect some way to configure a custom CA or maybe for docker-registry to read /var/run/secrets/kubernetes.io/serviceaccount/ca.crt which can be adjusted easily. Additional info: I tried setting the AWS_CA_BUNDLE environment variable for my docker-registry per this documentation: https://docs.aws.amazon.com/sdk-for-go/api/aws/session/ . However, setting it did not help.
There's no first class way to do it today (i've added this card to track it in our backlog: https://trello.com/c/ybOIFAOE). You should be able to construct a configmap containing the certs you want, and then manually edit the registry DC to mount them, however.
Here are some steps on using a configmap as a workaround: # oc rsync docker-registry-xxxx:/etc/pki/tls/certs/ . # cat My-Root-CA.crt >> ca-bundle.crt # cat My-Root-CA.crt >> ca-bundle.trust.crt # oc create configmap custom-ca-bundle --from-file=ca-bundle.crt --from-file=ca-bundle.trust.crt # oc volume dc/docker-registry --add --name custom-ca-bundle-vol -t configmap --configmap-name=custom-ca-bundle -m /etc/pki/tls/certs/
@bparees, I think there are two separate issues in this bz now. I'll leave it up to you on whether or not a new bz should be created. 1) Inability to easily override/append to /etc/pki/tls/certs 2) AWS_CA_BUNDLE environment variable not being picked up as expected. This is was I was originally reporting this BZ on. Issue #1 is good-to-go with https://trello.com/c/ybOIFAOE. Issue #2 is still not addressed but I've found a solution but only for 3.9: # oc rollout pause dc/docker-registry # oc create secret generic my-root-ca --from-file=my-root-ca=my-root-ca.crt # oc volume dc/docker-registry --add --overwrite --name=my-root-ca --mount-path=/etc/pki/tls/certs/verify -t secret --secret-name=my-root-ca # oc set env dc/docker-registry AWS_SDK_LOAD_CONFIG=true # oc set env dc/docker-registry AWS_CA_BUNDLE=/etc/pki/tls/certs/verify/rh-it-root-ca # oc rollout resume dc/docker-registry Steps above will *not* work for anything less than OCP 3.9 because that's when the AWS_CA_BUNDLE was added. References: [0] OCP 3.9: https://github.com/openshift/origin/blob/561755cee420f2c17f79928f608932cd9ee08014/vendor/github.com/aws/aws-sdk-go/aws/session/env_config.go#L150-L182 [1] OCP 3.7: https://github.com/openshift/origin/blob/a62384df654260bb4d6cd7f91946b0250876b503/vendor/github.com/aws/aws-sdk-go/aws/session/env_config.go#L124-L154
Seems like issue #2 would be solved by a solution to issue #1, no? If you were able to add the CA to the "system" trust, then you wouldn't need to set the AWS* env vars to pick it up from somewhere else.
> Seems like issue #2 would be solved by a solution to issue #1, no? Correct, the default behavior of the AWS SDK used by the s3 driver pulls from default CA bundle (i.e. /etc/pki/tls/...) according to the docs: https://docs.aws.amazon.com/sdk-for-go/api/aws/session/ > If you were able to add the CA to the "system" trust, then you wouldn't need to set the AWS* env vars to pick it up from somewhere else. Correct, however, using the AWS environment variable prevents OpenShift admins from having to maintain the ca-bundle configmap/secret in the event of a big CA (e.g. DigiCert) certificate revocation or addition.
With the introduction of OpenShift 4, Red Hat has delivered or roadmapped a substantial number of features based on feedback by our customers. Many of the enhancements encompass specific RFEs which have been requested, or deliver a comparable solution to a customer problem, rendering an RFE redundant. This bz (RFE) has been identified as a feature request not yet planned or scheduled for an OpenShift release and is being closed. If this feature is still an active request that needs to be tracked, Red Hat Support can assist in filing a request in the new JIRA RFE system, as well as provide you with updates as the RFE progress within our planning processes. Please open a new support case: https://access.redhat.com/support/cases/#/case/new Opening a New Support Case: https://access.redhat.com/support/cases/#/case/new As the new Jira RFE system is not yet public, Red Hat Support can help answer your questions about your RFEs via the same support case system.