Bug 1716835
| Summary: | Cluster deployed to AWS does not trust its internal docker registry, when trying to create application pushed to it, through web console | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Mikhail Krutov <mkrutov> | ||||
| Component: | ImageStreams | Assignee: | Ricardo Maraschini <rmarasch> | ||||
| Status: | CLOSED ERRATA | QA Contact: | XiuJuan Wang <xiuwang> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 4.1.0 | CC: | adam.kaplan, aos-bugs, bparees, deads, erich, jokerman, mfojtik, mmccomas, mtoth, obulatov, rmarasch, veer, wzheng | ||||
| Target Milestone: | --- | Keywords: | NeedsTestCase | ||||
| Target Release: | 4.4.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2020-05-13 21:51:46 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Mikhail Krutov
2019-06-04 09:07:44 UTC
You should use image stream tags to deploy images from the integrated registry. Why do you want to deploy it through the router? There is no particular reason to deploy it through router. I don't really care if external route address won't be available for internal deployment. However, as it creates image-registry.openshift-image-registry.svc by-default I'd expect it to work in default configuration. Especially since it works through oc command (oc new-app from template that uses image-registry.openshift-image-registry.svc:5000/some/image, as well as just oc new-app --docker-image=image-registry.openshift-image-registry.svc:5000/some/image), but fails to work through admin web console. $ oc tag image-registry.openshift-image-registry.svc:5000/default/foo@sha256:eb1d49bbd28d0179cec134433cb376341c1c34d798b30095064925e2f5f55f56 default/bar:latest
Tag bar:latest set to image-registry.openshift-image-registry.svc:5000/default/foo@sha256:eb1d49bbd28d0179cec134433cb376341c1c34d798b30095064925e2f5f55f56.
$ oc get is bar -o yaml
...
lastTransitionTime: "2019-06-04T13:17:25Z"
message: 'Internal error occurred: Get https://image-registry.openshift-image-registry.svc:5000/v2/:
x509: certificate signed by unknown authority'
reason: InternalError
status: "False"
type: ImportSuccess
...
The master API doesn't trust the certificate that was generated by service-ca.
As a workaround for this you can use https://github.com/openshift/api/blob/406574a7aa1eb0e916f4029433e5c28c6a53099b/config/v1/types_image.go#L41-L46 to set an additional CA to be trusted by the imagestream importer. You'd need to set the service CA for the cluster there. But longer term this should be fixed by having the apiserver mount and use the service-ca automatically, at least when doing imagestream imports. Teams own plumbing their own features. This is an imagestream import feature, so we'd expect plumbing from devex for all of it. See examples of plumbing configmaps from the old auth team here https://github.com/openshift/cluster-kube-apiserver-operator/blob/master/pkg/operator/configobservation/auth/auth_metadata.go#L82-L93 and what I think is the actual spot that you wired through for this before: https://github.com/openshift/cluster-openshift-apiserver-operator/blob/2ee3105a8a16eda652a503fbad6f68c4a08d6629/pkg/operator/workloadcontroller/workload_controller_openshiftapiserver_v311_00.go#L241-L273 . Attaching https://access.redhat.com/solutions/4395891 to denote the workaround for this bug. This remains an issue for OCP 4.3. With the CA-related work done in 4.2, fixing should be easier to implement. When it comes to version 4.3 these are my findings:
Pre reqs:
1. Expose internal image registry(default route).
2. Create a new project.
3. Push an image to the internal registry.
Deploying:
1. Does work with a Deployment[1] using internal registry URL
2. Does not work with a Deployment[2] using registry's defaultRoute[2]
2.1 Without tuning any configuration, pod reports:
ErrImagePull: Get 'default-route': x509: certificate signed by unknown authority
2.2 If added defaultRoute url to insecureRegistries(image.config.openshift.io), pod reports:
ImagePullBackOff: "unauthorized: authentication required"
Image tagging:
```
oc tag image-registry.openshift-image-registry.svc:5000/ns/img@<sha256> ns/img2:latest
```
Tagging succeds with no error but when we inspect the resulting image stream we can see:
```
$ oc get -o yaml is/hi2
...
status:
...
- conditions:
- message: 'Internal error occurred: image-registry.openshift-image-registry.svc:5000/myapp/hi@sha256:50c04bf6b70063dffc6001b5ec7d13afeb862189a26387988e4c682a97590312:
Get https://image-registry.openshift-image-registry.svc:5000/v2/: x509: certificate
signed by unknown authority'
reason: InternalError
type: ImportSuccess
```
[1] deploy.yaml using internal:
```
apiVersion: apps/v1
kind: Deployment
metadata:
name: hi
namespace: myapp
spec:
selector:
matchLabels:
app: hi
replicas: 1
template:
metadata:
labels:
app: hi
spec:
containers:
- name: hi
image: image-registry.openshift-image-registry.svc:5000/myapp/hi:latest
ports:
- containerPort: 8181
```
[2] deploy.yaml using external:
```
apiVersion: apps/v1
kind: Deployment
metadata:
name: hi
namespace: myapp
spec:
selector:
matchLabels:
app: hi
replicas: 1
template:
metadata:
labels:
app: hi
spec:
containers:
- name: hi
image: default-route-openshift-image-registry.apps.rmarasch-cluster-87.devcluster.openshift.com/myapp/hi:latest
ports:
- containerPort: 8181
```
Moving to 4.4.0 Test with 4.4.0-0.ci-2019-12-14-210519, could create deploy from webconsole with internal registry image, and the imagestream imported without x509 error.
$ oc get is ruby-hello-world-test -o yaml
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
annotations:
openshift.io/image.dockerRepositoryCheck: "2019-12-17T05:51:05Z"
creationTimestamp: "2019-12-17T05:51:04Z"
generation: 2
labels:
app: ruby-hello-world-test
app.kubernetes.io/component: ruby-hello-world-test
app.kubernetes.io/instance: ruby-hello-world-test
app.kubernetes.io/part-of: ruby-hello-world-app
name: ruby-hello-world-test
namespace: wxj
resourceVersion: "99616"
selfLink: /apis/image.openshift.io/v1/namespaces/wxj/imagestreams/ruby-hello-world-test
uid: 2414b93b-4fd4-499e-ab25-c8dd989fc923
spec:
lookupPolicy:
local: false
tags:
- annotations:
openshift.io/generated-by: OpenShiftWebConsole
openshift.io/imported-from: image-registry.openshift-image-registry.svc:5000/wxj/ruby-hello-world:latest
from:
kind: DockerImage
name: image-registry.openshift-image-registry.svc:5000/wxj/ruby-hello-world:latest
generation: 2
importPolicy: {}
name: latest
referencePolicy:
type: Source
status:
dockerImageRepository: image-registry.openshift-image-registry.svc:5000/wxj/ruby-hello-world-test
tags:
- items:
- created: "2019-12-17T05:51:05Z"
dockerImageReference: image-registry.openshift-image-registry.svc:5000/wxj/ruby-hello-world@sha256:d77f7052f673ec90a298ed52602ffe4959889ea1dfd02a6ebe26f7d4c1c7ea36
generation: 2
image: sha256:d77f7052f673ec90a298ed52602ffe4959889ea1dfd02a6ebe26f7d4c1c7ea36
tag: latest
$ oc get pods
NAME READY STATUS RESTARTS AGE
ruby-hello-world-1-build 0/1 Completed 0 10m
ruby-hello-world-test-1-deploy 0/1 Completed 0 5m31s
ruby-hello-world-test-1-w8klj 1/1 Running 0 5m19s
*** Bug 1796749 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581 |