Bug 1653166 - SmartState Analysis fails with registry.redhat.io which now requires authentication for OCP 3.11 [NEEDINFO]
Summary: SmartState Analysis fails with registry.redhat.io which now requires authenti...
Keywords:
Status: POST
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Documentation
Version: 5.9.5
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: GA
: 5.10.12
Assignee: Suyog Sainkar
QA Contact: Red Hat CloudForms Documentation
Red Hat CloudForms Documentation
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-26 06:40 UTC by Alan C
Modified: 2020-02-20 03:19 UTC (History)
16 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Category: ---
Cloudforms Team: Container Management
Target Upstream Version:
ssainkar: needinfo? (cben)


Attachments (Terms of Use)

Description Alan C 2018-11-26 06:40:52 UTC
Description of problem:

SmartState Analysis container image scanning fails with registry.redhat.io which now requires authentication on OCP 3.11.

Using openshift-management playbook CFME 4.6 is deployed inside OCP 3.11.

When scanning any container images, it fails with this from the manageiq-img-scan-XXXX pod in the management-infra namespace:

[root@master1 ~]# oc logs -f manageiq-img-scan-26d02 
2018/11/26 05:57:34 Pulling image registry.redhat.io/dotnet/dotnetcore-10-rhel7@sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf
2018/11/26 05:57:34 Pulling image with authentication Default Empty Authentication failed: Get https://registry.redhat.io/v2/dotnet/dotnetcore-10-rhel7/manifests/sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531
2018/11/26 05:57:35 Pulling image with authentication /var/run/secrets/kubernetes.io/inspector-admin-secret-inspector-admin-dockercfg-xtv4h/.dockercfg/172.30.116.229:5000 failed: Get https://registry.redhat.io/v2/dotnet/dotnetcore-10-rhel7/manifests/sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531
2018/11/26 05:57:35 Pulling image with authentication /var/run/secrets/kubernetes.io/inspector-admin-secret-inspector-admin-dockercfg-xtv4h/.dockercfg/docker-registry.default.svc.cluster.local:5000 failed: Get https://registry.redhat.io/v2/dotnet/dotnetcore-10-rhel7/manifests/sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531
2018/11/26 05:57:35 Pulling image with authentication /var/run/secrets/kubernetes.io/inspector-admin-secret-inspector-admin-dockercfg-xtv4h/.dockercfg/docker-registry.default.svc:5000 failed: Get https://registry.redhat.io/v2/dotnet/dotnetcore-10-rhel7/manifests/sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531
2018/11/26 05:57:35 !!!WARNING!!! It is insecure to serve the image content without setting
2018/11/26 05:57:35 an auth token. Please set INSPECTOR_AUTH_TOKEN in your environment.
2018/11/26 05:57:35 Serving image content on webdav://0.0.0.0:8080/api/v1/content/

The pod spec:

[root@master1 ~]# oc get pod manageiq-img-scan-26d02 -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    manageiq.org/guid: 0c9c5c74-f84a-48d1-9ad7-3449b9888c8d
    manageiq.org/hostname: cloudforms-0
    manageiq.org/image: registry.redhat.io/dotnet/dotnetcore-10-rhel7@sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf
    manageiq.org/jobid: 26d02975-e147-491a-9959-f7648a30399d
    openshift.io/scc: privileged
  creationTimestamp: 2018-11-26T05:57:30Z
  labels:
    manageiq.org: "true"
    name: manageiq-img-scan-26d02
  name: manageiq-img-scan-26d02
  namespace: management-infra
  resourceVersion: "266684"
  selfLink: /api/v1/namespaces/management-infra/pods/manageiq-img-scan-26d02
  uid: 2928e70c-f140-11e8-95cf-0228184d2b86
spec:
  containers:
  - command:
    - /usr/bin/image-inspector
    - --chroot
    - --image=registry.redhat.io/dotnet/dotnetcore-10-rhel7@sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf
    - --scan-type=openscap
    - --serve=0.0.0.0:8080
    - --dockercfg=/var/run/secrets/kubernetes.io/inspector-admin-secret-inspector-admin-dockercfg-xtv4h/.dockercfg
    - --cve-url=https://www.redhat.com/security/data/metrics/ds/
    image: registry.redhat.io/openshift3/image-inspector:latest
    imagePullPolicy: Always
    name: image-inspector
[...trimmed...]

Appears that inspector-admin-dockercfg-xxx doesn't appear to contain registry.redhat.io authentication info even the OCP cluster is built with oreg_auth_user/password info in the inventory.

Checked that /var/lib/origin/.docker/config.json on all nodes in the cluster all contain proper registry.redhat.io auth info.

However, based on github manageiq-providers-kubernetes/blob/master/app/models/manageiq/providers/kubernetes/container_manager/scanning/job.rb, it would include any additional imagepullsecret it finds under inspector-admin.

Thus, appears the following workaround I figured out could help...

Since openshift-ansible creates a proper dockercfg on root on each node by default with the registry.redhat.io auth info already, that can be used:

[root@master1 ~]# oc create secret generic redhat-io \
> --from-file=.dockercfg=.docker/config.json \
> --type=kubernetes.io/dockercfg
secret/redhat-io created

[root@master1 ~]# oc secrets link inspector-admin redhat-io --for=mount,pull
[root@master1 ~]# oc describe sa inspector-admin
Name:                inspector-admin
Namespace:           management-infra
Labels:              <none>
Annotations:         <none>
Image pull secrets:  inspector-admin-dockercfg-xtv4h
                     redhat-io
Mountable secrets:   inspector-admin-dockercfg-xtv4h
                     inspector-admin-token-hwdt8
                     redhat-io
Tokens:              inspector-admin-token-hwdt8
                     inspector-admin-token-jcjff
Events:              <none>

Now let's rescan and it appears to work fine:

[root@master1 ~]# oc logs -f manageiq-img-scan-2833d 
2018/11/26 06:09:48 Pulling image registry.redhat.io/dotnet/dotnetcore-10-rhel7@sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf
2018/11/26 06:09:49 Pulling image with authentication /var/run/secrets/kubernetes.io/inspector-admin-secret-inspector-admin-dockercfg-xtv4h/.dockercfg/docker-registry.default.svc:5000 failed: Get https://registry.redhat.io/v2/dotnet/dotnetcore-10-rhel7/manifests/sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531
2018/11/26 06:10:00 Downloading Image (238456Kb downloaded)
2018/11/26 06:10:07 Finished Downloading Image (238456Kb downloaded)
2018/11/26 06:10:07 Extracting image registry.redhat.io/dotnet/dotnetcore-10-rhel7@sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf to /var/tmp/image-inspector-993186236
2018/11/26 06:10:21 Writing OpenSCAP results to /var/tmp/image-inspector-scan-results-090078699
2018/11/26 06:10:32 !!!WARNING!!! It is insecure to serve the image content without setting
2018/11/26 06:10:32 an auth token. Please set INSPECTOR_AUTH_TOKEN in your environment.
2018/11/26 06:10:32 Serving image content on webdav://0.0.0.0:8080/api/v1/content/

You can see it still complains about the inpector-admin-dockercfg doesn't contain the needed auth info. But you can see the pod spec provides the second dockercfg on for image-inspector command, which allows it to pull the image that it's trying to scan.

[root@bastion ~]# oc get pod manageiq-img-scan-2833d -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    manageiq.org/guid: 0c9c5c74-f84a-48d1-9ad7-3449b9888c8d
    manageiq.org/hostname: cloudforms-0
    manageiq.org/image: registry.redhat.io/dotnet/dotnetcore-10-rhel7@sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf
    manageiq.org/jobid: 2833db6d-f421-4641-b1dd-b3117ac4b636
    openshift.io/scc: privileged
  creationTimestamp: 2018-11-26T06:09:45Z
  labels:
    manageiq.org: "true"
    name: manageiq-img-scan-2833d
  name: manageiq-img-scan-2833d
  namespace: management-infra
  resourceVersion: "268899"
  selfLink: /api/v1/namespaces/management-infra/pods/manageiq-img-scan-2833d
  uid: df3cc130-f141-11e8-95cf-0228184d2b86
spec:
  containers:
  - command:
    - /usr/bin/image-inspector
    - --chroot
    - --image=registry.redhat.io/dotnet/dotnetcore-10-rhel7@sha256:0a8f88ca46fd7ef467a61fadcc46159676a4271527a54466372b642ba2af9eaf
    - --scan-type=openscap
    - --serve=0.0.0.0:8080
    - --dockercfg=/var/run/secrets/kubernetes.io/inspector-admin-secret-inspector-admin-dockercfg-xtv4h/.dockercfg
    - --dockercfg=/var/run/secrets/kubernetes.io/inspector-admin-secret-redhat-io/.dockercfg
    - --cve-url=https://www.redhat.com/security/data/metrics/ds/
    image: registry.redhat.io/openshift3/image-inspector:latest
    imagePullPolicy: Always
    name: image-inspector
[...trimmed...]


Version-Release number of selected component (if applicable):

CFME 4.6 on OCP 3.11 using image-inspector:latest


How reproducible:


Steps to Reproduce:
1. Perform a scan on a container image from CFME web console.
2. Look at the log of manageiq-img-scan-xxx pod.
3. Check openscap report/info from CFME web console on that container image.

Actual results:

manageiq-img-scan-xxx fails and no openscap report/info.


Expected results:

manageiq-img-scan-xxx successful and shows openscap report/info.

Additional info:

So something needs to be done to address registry.redhat.io now require authentication?

Comment 4 Beni Paskin-Cherniavsky 2018-12-04 14:48:20 UTC
Thanks for the details!
AFAICT *technically* this is not a bug, just needs documentation.

That said, the customer experience is not good — need special convoluted actions just to use several Red Hat products together :-(

https://access.redhat.com/documentation/en-us/red_hat_cloudforms/4.7-Beta/html-single/scanning_container_images_in_cloudforms_with_openscap/
doesn't explain anything about pull secrets.
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/199, bz 1529510 added multiple secrets support, but didn't seem to have been documented.
"If the "inspector-admin" service account can pull images from the secure registry then image-inspector will be to pull those images."

> Checked that /var/lib/origin/.docker/config.json on all nodes in the cluster all contain proper registry.redhat.io auth info.

Interesting.  By design image-inspector does not (can not) rely on kubelet's ability to pull the image for other pods.  It currently does rely on docker daemon to pull, so maybe this should have worked (?)
(The plan was to make image-inspector switch to pulling itself with libcontainer, at which point machine-wide config would certainly not help.
https://github.com/openshift/image-inspector/pull/95
This probably won't land, due to lack of resources to actively develop image-inspector.)

> Appears that inspector-admin-dockercfg-xxx doesn't appear to contain registry.redhat.io authentication info even the OCP cluster is built with oreg_auth_user/password info in the inventory.

Is this a standard way to configure clusters with access to registry.redhat.io?
Looks so, according to
https://docs.okd.io/latest/install_config/configuring_red_hat_registry.html
it says there should also be an ImageStreamSecret — we should explore whether image scanning could use that.


Note You need to log in before you can comment on or make changes to this bug.