Bug 2060610 - Broken access to public images: Unable to connect to the server: no basic auth credentials
Summary: Broken access to public images: Unable to connect to the server: no basic aut...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.10
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.10.0
Assignee: Oleg Bulatov
QA Contact: Mike Fiedler
URL:
Whiteboard:
Depends On: 2060605
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-03-03 20:42 UTC by OpenShift BugZilla Robot
Modified: 2022-03-10 16:44 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-10 16:44:40 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift image-registry pull 317 0 None open [release-4.10] Bug 2060610: Revert "Fixed incorrect access denied error logs" 2022-03-03 20:42:20 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:44:58 UTC

Description OpenShift BugZilla Robot 2022-03-03 20:42:06 UTC
+++ This bug was initially created as a clone of Bug #2060605 +++

Bug 1902456's [1] seems to have broken access to public images.  Testing in a 4.10.0 cluster:

  $ oc get -o jsonpath='{.status.desired.version}{"\n"}' clusterversion version
  4.10.0

Exposing the registry, per [2]:

  $ oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge
  $ HOST=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')

Grant public access to the openshift namespace (do we doc this somewhere):

  $ oc adm policy add-role-to-group system:image-puller system:unauthenticated --namespace openshift
  Warning: Group 'system:unauthenticated' not found
clusterrole.rbac.authorization.k8s.io/system:image-puller added: "system:unauthenticated"
  $ oc adm policy add-role-to-group system:image-puller system:authenticated --namespace openshift
Warning: Group 'system:authenticated' not found
clusterrole.rbac.authorization.k8s.io/system:image-puller added: "system:authenticated"

Try to fetch image metadata:

  $ oc image info --insecure "${HOST}/openshift/tools"
  Unable to connect to the server: no basic auth credentials


For comparison, here is a cluster-bot cluster built from [3] (reverting [1]):

  $ oc get -o jsonpath='{.status.desired.version}{"\n"}' clusterversion version
  4.10.0-0.ci.test-2022-03-03-194357-ci-ln-dmrzjkb-latest
  $ oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge
  $ HOST=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')
  $ oc adm policy add-role-to-group system:image-puller system:unauthenticated --namespace openshift
  $ oc adm policy add-role-to-group system:image-puller system:authenticated --namespace openshift
  $ oc image info --insecure "${HOST}/openshift/tools:latest"
  Name:        default-route-openshift-image-registry.apps.ci-ln-dmrzjkb-72292.origin-ci-int-gce.dev.rhcloud.com/openshift/tools:latest
  Digest:      sha256:3b5588eec3e985d73f246bc80dc618cff889da5ee652863fe750851e3e26dc09
  ...

[1]: https://github.com/openshift/image-registry/pull/291
[2]: https://docs.openshift.com/container-platform/4.9/registry/securing-exposing-registry.html#registry-exposing-default-registry-manually_securing-exposing-registry
[3]: https://github.com/openshift/image-registry/pull/315

--- Additional comment from wking on 2022-03-03 20:34:17 UTC ---

(In reply to W. Trevor King from comment #0)
>   $ oc adm policy add-role-to-group system:image-puller system:authenticated --namespace openshift

This grant is irrelevant for our unauthenticated test, and I get the same '' after rolling it back:

  $ oc adm policy remove-role-from-group system:image-puller system:unauthenticated --namespace openshift
  clusterrole.rbac.authorization.k8s.io/system:image-puller removed: "system:unauthenticated"
  $ oc image info --insecure "${HOST}/openshift/tools"
  Unable to connect to the server: no basic auth credentials

So simple reproducer is:

  oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge
  oc adm policy add-role-to-group system:image-puller system:unauthenticated --namespace openshift
  sleep 4  # not sure, must need some time to let the registry operator create the Route
  HOST=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')
  oc image info --insecure "${HOST}/openshift/tools"

And you should see a successful fetch, but instead 4.10.0 gives 'no basic auth credentials'

--- Additional comment from deads on 2022-03-03 20:38:52 UTC ---

Marking blocker+ for 4.10.0.

This bug prevents anonymous image pulls from the internal openshift image-registry.

1. This is a regression from 4.9 behavior.
2. There is no workaround.
3. We have no way of knowing if clusters are impacted before they upgrade
4. image-registry pulls are critical path on containerized workloads.  We don't have data on frequency of anonymous pulls, but we have no reason to believe it is rare.  Many (most?) images are not sensitive.

Comment 1 W. Trevor King 2022-03-03 21:34:48 UTC
Setting blocker+ in this 4.10.0 bug, per [1].

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=2060605#c2

Comment 2 Mike Fiedler 2022-03-03 21:40:21 UTC
Verified with the simple reproducer

[mifiedle@mffiedler ~]$ oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge                                                                         
config.imageregistry.operator.openshift.io/cluster patched                                                                                                                                                         
[mifiedle@mffiedler ~]$ oc adm policy add-role-to-group system:image-puller system:unauthenticated --namespace openshift                                                                                           
Warning: Group 'system:unauthenticated' not found                                                                                                                                                                  
clusterrole.rbac.authorization.k8s.io/system:image-puller added: "system:unauthenticated"
[mifiedle@mffiedler ~]$ HOST=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')
[mifiedle@mffiedler ~]$ oc image info --insecure "${HOST}/openshift/tools"
W0303 16:39:27.843335   48196 helpers.go:151] Defaulting of registry auth file to "${HOME}/.docker/config.json" is deprecated. The default will be switched to podman config locations in the future version.
Name:        default-route-openshift-image-registry.apps.ci-ln-jvfxyht-76ef8.origin-ci-int-aws.dev.rhcloud.com/openshift/tools:latest
Digest:      sha256:3b5588eec3e985d73f246bc80dc618cff889da5ee652863fe750851e3e26dc09
Media Type:  application/vnd.docker.distribution.manifest.v2+json
Created:     15d ago
Image Size:  273.7MB in 7 layers
Layers:      83.3MB  sha256:eac1b95df832dc9f172fd1f07e7cb50c1929b118a4249ddd02c6318a677b506a
             1.789kB sha256:47aa3ed2034c4f27622b989b26c06087de17067268a19a1b3642a7e2686cd1a3
             16.81MB sha256:6a15bb59e41f721e4315ae053d006cfcdcf06bff2bd7586dacff36bf0272c30a
             11.6MB  sha256:996635267df8ca4c548dc4d34d06d19c54e9fc37dfc703a1058c2aa2e95bfd06
             485.5kB sha256:81b704cb35c4fdab7096f5284395de67f8b9d163f69777c37e3e40a86aeaf238
             48.77MB sha256:2edd04531b637ddb36e15719002efb85c91edbc27b0bbd894bd97311ed47979d
             112.8MB sha256:7d1c2db5ae0d6e6acbd2027066bc4e17a6e41d1cc0e4831b8a167ca05ef53ce2
OS:          linux
Arch:        amd64
Command:     /usr/bin/bash
User:        0
Environment: __doozer=merge
             BUILD_RELEASE=202202081751.p0.g544601e.assembly.stream
             BUILD_VERSION=v4.10.0
             OS_GIT_MAJOR=4
             OS_GIT_MINOR=10


<snip - rest of output deleted>

Comment 7 errata-xmlrpc 2022-03-10 16:44:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.