Created attachment 1576958 [details]
Example output (screenshot)
Description of problem:
Cluster has a internal docker registry configured by-default. If I push an image to internal docker registry, then try to deploy said image using internal URL, I get Unknown CA Authority errors.
This behavior is different through OC command line utility, where it just prints a warning about internal reigstry being insecure.
If I add internal registry to the configuration using
> $ oc edit image.config.openshift.io
then nothing changes, despite worker nodes having properly configured /etc/containers/registries.conf:
registries = ["registry.access.redhat.com", "docker.io"]
registries = ["image-registry.openshift-image-registry.svc:5000", "registry-openshift-image-registry.apps.mkrutov.openshift-aws.rhocf-dev.com"]
registries = 
Version-Release number of selected component (if applicable):
100% every time
Steps to Reproduce:
1. Install fresh openshift cluster to aws using openshift-installer
2. Expose docker registry
3. Push image to docker registry
4. Try to deploy image using openshift console.
It fails with x509 Unknown Authority error.
It deploys a docker image that was pushed to internal registry.
I think that by-default openshift should trust its internal components.
You should use image stream tags to deploy images from the integrated registry. Why do you want to deploy it through the router?
There is no particular reason to deploy it through router. I don't really care if external route address won't be available for internal deployment.
However, as it creates image-registry.openshift-image-registry.svc by-default I'd expect it to work in default configuration.
Especially since it works through oc command (oc new-app from template that uses image-registry.openshift-image-registry.svc:5000/some/image, as well as just oc new-app --docker-image=image-registry.openshift-image-registry.svc:5000/some/image), but fails to work through admin web console.
$ oc tag image-registry.openshift-image-registry.svc:5000/default/foo@sha256:eb1d49bbd28d0179cec134433cb376341c1c34d798b30095064925e2f5f55f56 default/bar:latest
Tag bar:latest set to image-registry.openshift-image-registry.svc:5000/default/foo@sha256:eb1d49bbd28d0179cec134433cb376341c1c34d798b30095064925e2f5f55f56.
$ oc get is bar -o yaml
message: 'Internal error occurred: Get https://image-registry.openshift-image-registry.svc:5000/v2/:
x509: certificate signed by unknown authority'
The master API doesn't trust the certificate that was generated by service-ca.
As a workaround for this you can use https://github.com/openshift/api/blob/406574a7aa1eb0e916f4029433e5c28c6a53099b/config/v1/types_image.go#L41-L46 to set an additional CA to be trusted by the imagestream importer. You'd need to set the service CA for the cluster there.
But longer term this should be fixed by having the apiserver mount and use the service-ca automatically, at least when doing imagestream imports.
Teams own plumbing their own features. This is an imagestream import feature, so we'd expect plumbing from devex for all of it. See examples of plumbing configmaps from the old auth team here https://github.com/openshift/cluster-kube-apiserver-operator/blob/master/pkg/operator/configobservation/auth/auth_metadata.go#L82-L93 and what I think is the actual spot that you wired through for this before: https://github.com/openshift/cluster-openshift-apiserver-operator/blob/2ee3105a8a16eda652a503fbad6f68c4a08d6629/pkg/operator/workloadcontroller/workload_controller_openshiftapiserver_v311_00.go#L241-L273 .
Attaching https://access.redhat.com/solutions/4395891 to denote the workaround for this bug.
This remains an issue for OCP 4.3. With the CA-related work done in 4.2, fixing should be easier to implement.
When it comes to version 4.3 these are my findings:
1. Expose internal image registry(default route).
2. Create a new project.
3. Push an image to the internal registry.
1. Does work with a Deployment using internal registry URL
2. Does not work with a Deployment using registry's defaultRoute
2.1 Without tuning any configuration, pod reports:
ErrImagePull: Get 'default-route': x509: certificate signed by unknown authority
2.2 If added defaultRoute url to insecureRegistries(image.config.openshift.io), pod reports:
ImagePullBackOff: "unauthorized: authentication required"
oc tag image-registry.openshift-image-registry.svc:5000/ns/img@<sha256> ns/img2:latest
Tagging succeds with no error but when we inspect the resulting image stream we can see:
$ oc get -o yaml is/hi2
- message: 'Internal error occurred: image-registry.openshift-image-registry.svc:5000/myapp/hi@sha256:50c04bf6b70063dffc6001b5ec7d13afeb862189a26387988e4c682a97590312:
Get https://image-registry.openshift-image-registry.svc:5000/v2/: x509: certificate
signed by unknown authority'
 deploy.yaml using internal:
- name: hi
- containerPort: 8181
 deploy.yaml using external:
- name: hi
- containerPort: 8181
Moving to 4.4.0
Test with 4.4.0-0.ci-2019-12-14-210519, could create deploy from webconsole with internal registry image, and the imagestream imported without x509 error.
$ oc get is ruby-hello-world-test -o yaml
- created: "2019-12-17T05:51:05Z"
$ oc get pods
NAME READY STATUS RESTARTS AGE
ruby-hello-world-1-build 0/1 Completed 0 10m
ruby-hello-world-test-1-deploy 0/1 Completed 0 5m31s
ruby-hello-world-test-1-w8klj 1/1 Running 0 5m19s
*** Bug 1796749 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.