Bug 1716835 - Cluster deployed to AWS does not trust its internal docker registry, when trying to create application pushed to it, through web console
Summary: Cluster deployed to AWS does not trust its internal docker registry, when try...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: ImageStreams
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.4.0
Assignee: Ricardo Maraschini
QA Contact: XiuJuan Wang
URL:
Whiteboard:
: 1796749 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-04 09:07 UTC by Mikhail Krutov
Modified: 2020-05-13 21:51 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-13 21:51:46 UTC
Target Upstream Version:


Attachments (Terms of Use)
Example output (screenshot) (27.22 KB, image/png)
2019-06-04 09:07 UTC, Mikhail Krutov
no flags Details


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-openshift-apiserver-operator pull 259 'None' closed Bug 1716835: Hooking Image Registry internal service certificates 2020-05-14 00:38:36 UTC
Red Hat Knowledge Base (Solution) 4395891 None None None 2019-09-05 19:52:21 UTC
Red Hat Product Errata RHBA-2020:0581 None None None 2020-05-13 21:51:48 UTC

Description Mikhail Krutov 2019-06-04 09:07:44 UTC
Created attachment 1576958 [details]
Example output (screenshot)

Description of problem:
Cluster has a internal docker registry configured by-default. If I push an image to internal docker registry, then try to deploy said image using internal URL, I get Unknown CA Authority errors.

This behavior is different through OC command line utility, where it just prints a warning about internal reigstry being insecure.

If I add internal registry to the configuration using 

> $ oc edit image.config.openshift.io

then nothing changes, despite worker nodes having properly configured /etc/containers/registries.conf:

[registries]
  [registries.search]
    registries = ["registry.access.redhat.com", "docker.io"]
  [registries.insecure]
    registries = ["image-registry.openshift-image-registry.svc:5000", "registry-openshift-image-registry.apps.mkrutov.openshift-aws.rhocf-dev.com"]
  [registries.block]
    registries = []


Version-Release number of selected component (if applicable):
4.1 release

How reproducible:
100% every time

Steps to Reproduce:
1. Install fresh openshift cluster to aws using openshift-installer
2. Expose docker registry
3. Push image to docker registry
4. Try to deploy image using openshift console.

Actual results:

It fails with x509 Unknown Authority error. 

Expected results:

It deploys a docker image that was pushed to internal registry.

Additional info:
I think that by-default openshift should trust its internal components.

Comment 1 Oleg Bulatov 2019-06-04 12:17:30 UTC
You should use image stream tags to deploy images from the integrated registry. Why do you want to deploy it through the router?

Comment 2 Mikhail Krutov 2019-06-04 12:33:06 UTC
There is no particular reason to deploy it through router. I don't really care if external route address won't be available for internal deployment.

However, as it creates image-registry.openshift-image-registry.svc by-default I'd expect it to work in default configuration. 

Especially since it works through oc command (oc new-app from template that uses image-registry.openshift-image-registry.svc:5000/some/image, as well as just oc new-app --docker-image=image-registry.openshift-image-registry.svc:5000/some/image), but fails to work through admin web console.

Comment 3 Oleg Bulatov 2019-06-04 13:25:31 UTC
$ oc tag image-registry.openshift-image-registry.svc:5000/default/foo@sha256:eb1d49bbd28d0179cec134433cb376341c1c34d798b30095064925e2f5f55f56 default/bar:latest
Tag bar:latest set to image-registry.openshift-image-registry.svc:5000/default/foo@sha256:eb1d49bbd28d0179cec134433cb376341c1c34d798b30095064925e2f5f55f56.
$ oc get is bar -o yaml
...
      lastTransitionTime: "2019-06-04T13:17:25Z"
      message: 'Internal error occurred: Get https://image-registry.openshift-image-registry.svc:5000/v2/:
        x509: certificate signed by unknown authority'
      reason: InternalError
      status: "False"
      type: ImportSuccess
...

The master API doesn't trust the certificate that was generated by service-ca.

Comment 7 Ben Parees 2019-07-29 13:33:40 UTC
As a workaround for this you can use https://github.com/openshift/api/blob/406574a7aa1eb0e916f4029433e5c28c6a53099b/config/v1/types_image.go#L41-L46 to set an additional CA to be trusted by the imagestream importer.  You'd need to set the service CA for the cluster there.

But longer term this should be fixed by having the apiserver mount and use the service-ca automatically, at least when doing imagestream imports.

Comment 9 David Eads 2019-07-29 17:41:34 UTC
Teams own plumbing their own features.  This is an imagestream import feature, so we'd expect plumbing from devex for all of it.  See examples of plumbing configmaps from the old auth team here https://github.com/openshift/cluster-kube-apiserver-operator/blob/master/pkg/operator/configobservation/auth/auth_metadata.go#L82-L93 and what I think is the actual spot that you wired through for this before: https://github.com/openshift/cluster-openshift-apiserver-operator/blob/2ee3105a8a16eda652a503fbad6f68c4a08d6629/pkg/operator/workloadcontroller/workload_controller_openshiftapiserver_v311_00.go#L241-L273 .

Comment 13 Eric Rich 2019-09-05 19:52:22 UTC
Attaching https://access.redhat.com/solutions/4395891 to denote the workaround for this bug.

Comment 14 Adam Kaplan 2019-11-12 13:50:06 UTC
This remains an issue for OCP 4.3. With the CA-related work done in 4.2, fixing should be easier to implement.

Comment 15 Ricardo Maraschini 2019-11-13 12:56:42 UTC
When it comes to version 4.3 these are my findings:

Pre reqs:
1. Expose internal image registry(default route).
2. Create a new project.
3. Push an image to the internal registry.


Deploying:

1. Does work with a Deployment[1] using internal registry URL
2. Does not work with a Deployment[2] using registry's defaultRoute[2]
    2.1 Without tuning any configuration, pod reports:
        ErrImagePull: Get 'default-route': x509: certificate signed by unknown authority
    2.2 If added defaultRoute url to insecureRegistries(image.config.openshift.io), pod reports:
        ImagePullBackOff: "unauthorized: authentication required"


Image tagging:

```
oc tag image-registry.openshift-image-registry.svc:5000/ns/img@<sha256> ns/img2:latest
```

Tagging succeds with no error but when we inspect the resulting image stream we can see:

```
$ oc get -o yaml is/hi2
...
status:
...
  - conditions:
    - message: 'Internal error occurred: image-registry.openshift-image-registry.svc:5000/myapp/hi@sha256:50c04bf6b70063dffc6001b5ec7d13afeb862189a26387988e4c682a97590312:
        Get https://image-registry.openshift-image-registry.svc:5000/v2/: x509: certificate
        signed by unknown authority'
      reason: InternalError
      type: ImportSuccess
```


[1] deploy.yaml using internal:
```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hi
  namespace: myapp
spec:
  selector:
    matchLabels:
      app: hi
  replicas: 1
  template:
    metadata:
      labels:
        app: hi
    spec:
      containers:
        - name: hi
          image: image-registry.openshift-image-registry.svc:5000/myapp/hi:latest
          ports:
            - containerPort: 8181
```

[2] deploy.yaml using external:
```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hi
  namespace: myapp
spec:
  selector:
    matchLabels:
      app: hi
  replicas: 1
  template:
    metadata:
      labels:
        app: hi
    spec:
      containers:
        - name: hi
          image: default-route-openshift-image-registry.apps.rmarasch-cluster-87.devcluster.openshift.com/myapp/hi:latest
          ports:
            - containerPort: 8181

```

Comment 16 Adam Kaplan 2019-11-22 19:07:36 UTC
Moving to 4.4.0

Comment 18 XiuJuan Wang 2019-12-17 05:58:44 UTC
Test with 4.4.0-0.ci-2019-12-14-210519, could create deploy from webconsole with internal registry image, and the imagestream imported without x509 error.

$ oc get  is  ruby-hello-world-test -o yaml 
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  annotations:
    openshift.io/image.dockerRepositoryCheck: "2019-12-17T05:51:05Z"
  creationTimestamp: "2019-12-17T05:51:04Z"
  generation: 2
  labels:
    app: ruby-hello-world-test
    app.kubernetes.io/component: ruby-hello-world-test
    app.kubernetes.io/instance: ruby-hello-world-test
    app.kubernetes.io/part-of: ruby-hello-world-app
  name: ruby-hello-world-test
  namespace: wxj
  resourceVersion: "99616"
  selfLink: /apis/image.openshift.io/v1/namespaces/wxj/imagestreams/ruby-hello-world-test
  uid: 2414b93b-4fd4-499e-ab25-c8dd989fc923
spec:
  lookupPolicy:
    local: false
  tags:
  - annotations:
      openshift.io/generated-by: OpenShiftWebConsole
      openshift.io/imported-from: image-registry.openshift-image-registry.svc:5000/wxj/ruby-hello-world:latest
    from:
      kind: DockerImage
      name: image-registry.openshift-image-registry.svc:5000/wxj/ruby-hello-world:latest
    generation: 2
    importPolicy: {}
    name: latest
    referencePolicy:
      type: Source
status:
  dockerImageRepository: image-registry.openshift-image-registry.svc:5000/wxj/ruby-hello-world-test
  tags:
  - items:
    - created: "2019-12-17T05:51:05Z"
      dockerImageReference: image-registry.openshift-image-registry.svc:5000/wxj/ruby-hello-world@sha256:d77f7052f673ec90a298ed52602ffe4959889ea1dfd02a6ebe26f7d4c1c7ea36
      generation: 2
      image: sha256:d77f7052f673ec90a298ed52602ffe4959889ea1dfd02a6ebe26f7d4c1c7ea36
    tag: latest

$ oc get  pods 
NAME                             READY   STATUS      RESTARTS   AGE
ruby-hello-world-1-build         0/1     Completed   0          10m
ruby-hello-world-test-1-deploy   0/1     Completed   0          5m31s
ruby-hello-world-test-1-w8klj    1/1     Running     0          5m19s

Comment 19 Oleg Bulatov 2020-02-02 21:14:28 UTC
*** Bug 1796749 has been marked as a duplicate of this bug. ***

Comment 21 errata-xmlrpc 2020-05-13 21:51:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.