Description of problem:

SmartState Analysis container image scanning fails with registry.redhat.io which now requires authentication on OCP 3.11.

Using openshift-management playbook CFME 4.6 is deployed inside OCP 3.11.

When scanning any container images, it fails with this from the manageiq-img-scan-XXXX pod in the management-infra namespace:

[root@master1 ~]# oc logs -f manageiq-img-scan-26d02 
2018/11/26 05:57:34 Pulling image registry.redhat.io/dotnet/dotnetcore-10-rhel7@sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf
2018/11/26 05:57:34 Pulling image with authentication Default Empty Authentication failed: Get https://registry.redhat.io/v2/dotnet/dotnetcore-10-rhel7/manifests/sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531
2018/11/26 05:57:35 Pulling image with authentication /var/run/secrets/kubernetes.io/inspector-admin-secret-inspector-admin-dockercfg-xtv4h/.dockercfg/172.30.116.229:5000 failed: Get https://registry.redhat.io/v2/dotnet/dotnetcore-10-rhel7/manifests/sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531
2018/11/26 05:57:35 Pulling image with authentication /var/run/secrets/kubernetes.io/inspector-admin-secret-inspector-admin-dockercfg-xtv4h/.dockercfg/docker-registry.default.svc.cluster.local:5000 failed: Get https://registry.redhat.io/v2/dotnet/dotnetcore-10-rhel7/manifests/sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531
2018/11/26 05:57:35 Pulling image with authentication /var/run/secrets/kubernetes.io/inspector-admin-secret-inspector-admin-dockercfg-xtv4h/.dockercfg/docker-registry.default.svc:5000 failed: Get https://registry.redhat.io/v2/dotnet/dotnetcore-10-rhel7/manifests/sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531
2018/11/26 05:57:35 !!!WARNING!!! It is insecure to serve the image content without setting
2018/11/26 05:57:35 an auth token. Please set INSPECTOR_AUTH_TOKEN in your environment.
2018/11/26 05:57:35 Serving image content on webdav://0.0.0.0:8080/api/v1/content/

The pod spec:

[root@master1 ~]# oc get pod manageiq-img-scan-26d02 -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    manageiq.org/guid: 0c9c5c74-f84a-48d1-9ad7-3449b9888c8d
    manageiq.org/hostname: cloudforms-0
    manageiq.org/image: registry.redhat.io/dotnet/dotnetcore-10-rhel7@sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf
    manageiq.org/jobid: 26d02975-e147-491a-9959-f7648a30399d
    openshift.io/scc: privileged
  creationTimestamp: 2018-11-26T05:57:30Z
  labels:
    manageiq.org: "true"
    name: manageiq-img-scan-26d02
  name: manageiq-img-scan-26d02
  namespace: management-infra
  resourceVersion: "266684"
  selfLink: /api/v1/namespaces/management-infra/pods/manageiq-img-scan-26d02
  uid: 2928e70c-f140-11e8-95cf-0228184d2b86
spec:
  containers:
  - command:
    - /usr/bin/image-inspector
    - --chroot
    - --image=registry.redhat.io/dotnet/dotnetcore-10-rhel7@sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf
    - --scan-type=openscap
    - --serve=0.0.0.0:8080
    - --dockercfg=/var/run/secrets/kubernetes.io/inspector-admin-secret-inspector-admin-dockercfg-xtv4h/.dockercfg
    - --cve-url=https://www.redhat.com/security/data/metrics/ds/
    image: registry.redhat.io/openshift3/image-inspector:latest
    imagePullPolicy: Always
    name: image-inspector
[...trimmed...]

Appears that inspector-admin-dockercfg-xxx doesn't appear to contain registry.redhat.io authentication info even the OCP cluster is built with oreg_auth_user/password info in the inventory.

Checked that /var/lib/origin/.docker/config.json on all nodes in the cluster all contain proper registry.redhat.io auth info.

However, based on github manageiq-providers-kubernetes/blob/master/app/models/manageiq/providers/kubernetes/container_manager/scanning/job.rb, it would include any additional imagepullsecret it finds under inspector-admin.

Thus, appears the following workaround I figured out could help...

Since openshift-ansible creates a proper dockercfg on root on each node by default with the registry.redhat.io auth info already, that can be used:

[root@master1 ~]# oc create secret generic redhat-io \
> --from-file=.dockercfg=.docker/config.json \
> --type=kubernetes.io/dockercfg
secret/redhat-io created

[root@master1 ~]# oc secrets link inspector-admin redhat-io --for=mount,pull
[root@master1 ~]# oc describe sa inspector-admin
Name:                inspector-admin
Namespace:           management-infra
Labels:              <none>
Annotations:         <none>
Image pull secrets:  inspector-admin-dockercfg-xtv4h
                     redhat-io
Mountable secrets:   inspector-admin-dockercfg-xtv4h
                     inspector-admin-token-hwdt8
                     redhat-io
Tokens:              inspector-admin-token-hwdt8
                     inspector-admin-token-jcjff
Events:              <none>

Now let's rescan and it appears to work fine:

[root@master1 ~]# oc logs -f manageiq-img-scan-2833d 
2018/11/26 06:09:48 Pulling image registry.redhat.io/dotnet/dotnetcore-10-rhel7@sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf
2018/11/26 06:09:49 Pulling image with authentication /var/run/secrets/kubernetes.io/inspector-admin-secret-inspector-admin-dockercfg-xtv4h/.dockercfg/docker-registry.default.svc:5000 failed: Get https://registry.redhat.io/v2/dotnet/dotnetcore-10-rhel7/manifests/sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531
2018/11/26 06:10:00 Downloading Image (238456Kb downloaded)
2018/11/26 06:10:07 Finished Downloading Image (238456Kb downloaded)
2018/11/26 06:10:07 Extracting image registry.redhat.io/dotnet/dotnetcore-10-rhel7@sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf to /var/tmp/image-inspector-993186236
2018/11/26 06:10:21 Writing OpenSCAP results to /var/tmp/image-inspector-scan-results-090078699
2018/11/26 06:10:32 !!!WARNING!!! It is insecure to serve the image content without setting
2018/11/26 06:10:32 an auth token. Please set INSPECTOR_AUTH_TOKEN in your environment.
2018/11/26 06:10:32 Serving image content on webdav://0.0.0.0:8080/api/v1/content/

You can see it still complains about the inpector-admin-dockercfg doesn't contain the needed auth info. But you can see the pod spec provides the second dockercfg on for image-inspector command, which allows it to pull the image that it's trying to scan.

[root@bastion ~]# oc get pod manageiq-img-scan-2833d -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    manageiq.org/guid: 0c9c5c74-f84a-48d1-9ad7-3449b9888c8d
    manageiq.org/hostname: cloudforms-0
    manageiq.org/image: registry.redhat.io/dotnet/dotnetcore-10-rhel7@sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf
    manageiq.org/jobid: 2833db6d-f421-4641-b1dd-b3117ac4b636
    openshift.io/scc: privileged
  creationTimestamp: 2018-11-26T06:09:45Z
  labels:
    manageiq.org: "true"
    name: manageiq-img-scan-2833d
  name: manageiq-img-scan-2833d
  namespace: management-infra
  resourceVersion: "268899"
  selfLink: /api/v1/namespaces/management-infra/pods/manageiq-img-scan-2833d
  uid: df3cc130-f141-11e8-95cf-0228184d2b86
spec:
  containers:
  - command:
    - /usr/bin/image-inspector
    - --chroot
    - --image=registry.redhat.io/dotnet/dotnetcore-10-rhel7@sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf
    - --scan-type=openscap
    - --serve=0.0.0.0:8080
    - --dockercfg=/var/run/secrets/kubernetes.io/inspector-admin-secret-inspector-admin-dockercfg-xtv4h/.dockercfg
    - --dockercfg=/var/run/secrets/kubernetes.io/inspector-admin-secret-redhat-io/.dockercfg
    - --cve-url=https://www.redhat.com/security/data/metrics/ds/
    image: registry.redhat.io/openshift3/image-inspector:latest
    imagePullPolicy: Always
    name: image-inspector
[...trimmed...]


Version-Release number of selected component (if applicable):

CFME 4.6 on OCP 3.11 using image-inspector:latest


How reproducible:


Steps to Reproduce:
1. Perform a scan on a container image from CFME web console.
2. Look at the log of manageiq-img-scan-xxx pod.
3. Check openscap report/info from CFME web console on that container image.

Actual results:

manageiq-img-scan-xxx fails and no openscap report/info.


Expected results:

manageiq-img-scan-xxx successful and shows openscap report/info.

Additional info:

So something needs to be done to address registry.redhat.io now require authentication?