Bug 1543122 - RBAC DENY (system:anonymous fails to imagestreams/layers.image.openshift.io)
Summary: RBAC DENY (system:anonymous fails to imagestreams/layers.image.openshift.io)
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.9.0
Assignee: Oleg Bulatov
QA Contact: Dongbo Yan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-02-07 18:47 UTC by Greg Rodriguez II
Modified: 2021-12-10 15:38 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: the signature importer tries to import signatures from the internal registry without credentials. Consequence: the registry checks if the anonymous user can get signatures using SAR requests. Fix: the signature importer skips the internal registry, because the internal registry and the signature importer work with the same storage. Result: no SAR requests.
Clone Of:
Environment:
Last Closed: 2019-05-09 15:21:01 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:0489 0 None None None 2018-03-28 14:26:55 UTC

Description Greg Rodriguez II 2018-02-07 18:47:44 UTC
Description of problem:
After upgrading from OCP 3.6 to 3.7 we see hundreds of messages within journald like the following (approximately one RBAC DENY error per image which physically resides within the internal Docker registry):

atomic-openshift-master-api[1878]: I0109 08:53:25.734369    1878 rbac.go:116] RBAC DENY: user "system:anonymous" groups ["system:unauthenticated"] cannot "get" resource "imagestreams/layers.image.openshift.io" named "testimage" in namespace "openshift"

Version-Release number of selected component (if applicable):
3.7

How reproducible:
Very, was able to reproduce with QuickLab following steps provided by customer

Steps to Reproduce:
To make sure we're testing as close to like/like I've provided a list of steps which will hopefully show what we're seeing.

Notes:
 - Please disable atomic-openshift-master-api and atomic-openshift-master-controllers on all nodes but a single instance.  This makes troubleshooting easier :)
 - Perform all remaining operations from the remaining fully operational openshift-master-api node
  --> Ensure you are running these commands as root unless otherwise specified in the directions

## Check loglevel of master api
ps auxwww|grep 'openshift start master api'|grep 'loglevel=2' && echo 'INFO: Loglevel is set' || 'ERROR: Loglevel not set to the correct value'

## Ensure that there aren't any lingering /root/.docker files/dir
rm -Rf /root/.docker

## Create docker group if it doesn't exist
groupadd docker

## Create local user
useradd testuser1 -G docker

## Restart docker daemon
systemctl restart docker

## Enable debug mode for your docker registry
oc set env dc docker-registry REGISTRY_LOG_LEVEL=debug -n default --config /etc/origin/master/admin.kubeconfig

## Scale the registry to a single replica (makes tracking logs much easier)
oc scale dc docker-registry --replicas=1 --config /etc/origin/master/admin.kubeconfig

## Create a new project
oc adm new-project testimages --config /etc/origin/master/admin.kubeconfig

## Pull down images
docker pull registry.access.redhat.com/rhel7-atomic:7.4-119
docker pull registry.access.redhat.com/rhel7-atomic:7.4-97
docker pull registry.access.redhat.com/rhel7-atomic:7.4-88
docker pull registry.access.redhat.com/rhel7-atomic:7.4-79
docker pull registry.access.redhat.com/rhel7-atomic:7.4-70
docker pull registry.access.redhat.com/rhel7-atomic:7.4-54
docker pull registry.access.redhat.com/rhel7-atomic:7.3-35
docker pull registry.access.redhat.com/rhel7-atomic:7.3-31
docker pull registry.access.redhat.com/rhel7-atomic:7.3-26
docker pull registry.access.redhat.com/rhel7-atomic:7.3-19
docker pull registry.access.redhat.com/rhel7-atomic:7.3-15
docker pull registry.access.redhat.com/rhel7-atomic:7.3-9
docker pull registry.access.redhat.com/rhel7-atomic:7.3-6

export DOCKERREG=$(oc get svc -n default --no-headers --config /etc/origin/master/admin.kubeconfig|grep docker-registry|awk '{print $2}')
echo "export DOCKERREG=${DOCKERREG}" >> /home/testuser1/.bash_profile

## Tag Docker images
docker tag a9b2247c0747 ${DOCKERREG}:5000/testimages/myimage1:latest
docker tag 9d66e4acd0bb ${DOCKERREG}:5000/testimages/myimage2:latest
docker tag dfbd36ddc7ca ${DOCKERREG}:5000/testimages/myimage3:latest
docker tag 6b5f2e218721 ${DOCKERREG}:5000/testimages/myimage4:latest
docker tag 0fec25183884 ${DOCKERREG}:5000/testimages/myimage5:latest
docker tag 68a151d90449 ${DOCKERREG}:5000/testimages/myimage6:latest
docker tag 1d81b617b801 ${DOCKERREG}:5000/testimages/myimage7:latest
docker tag c921f9911a21 ${DOCKERREG}:5000/testimages/myimage8:latest
docker tag b2a215996d7a ${DOCKERREG}:5000/testimages/myimage9:latest
docker tag 99af91042675 ${DOCKERREG}:5000/testimages/myimage10:latest
docker tag 8f0f6a9eb42d ${DOCKERREG}:5000/testimages/myimage11:latest
docker tag c8833d5bce74 ${DOCKERREG}:5000/testimages/myimage12:latest
docker tag fb7880540f48 ${DOCKERREG}:5000/testimages/myimage13:latest

## Switch users account to your testuser1
sudo -i -u testuser1 

## Login to OpenShift with a user who can push images into our custom namespace 'testimages'
oc login -u .......

## Login to your internal registry
docker login -u empty -p $(oc whoami -t) ${DOCKERREG}:5000

## Push your images into the internal registry
docker push ${DOCKERREG}:5000/testimages/myimage1:latest
docker push ${DOCKERREG}:5000/testimages/myimage2:latest
docker push ${DOCKERREG}:5000/testimages/myimage3:latest
docker push ${DOCKERREG}:5000/testimages/myimage4:latest
docker push ${DOCKERREG}:5000/testimages/myimage5:latest
docker push ${DOCKERREG}:5000/testimages/myimage6:latest
docker push ${DOCKERREG}:5000/testimages/myimage7:latest
docker push ${DOCKERREG}:5000/testimages/myimage8:latest
docker push ${DOCKERREG}:5000/testimages/myimage9:latest
docker push ${DOCKERREG}:5000/testimages/myimage10:latest
docker push ${DOCKERREG}:5000/testimages/myimage11:latest
docker push ${DOCKERREG}:5000/testimages/myimage12:latest
docker push ${DOCKERREG}:5000/testimages/myimage13:latest

## Switch back to root
exit

## Stop our services
systemctl stop atomic-openshift-master-controllers
systemctl stop atomic-openshift-master-api

## Wait a minute or so
sleep 30

## Start everything back up again
systemctl start atomic-openshift-master-api
systemctl start atomic-openshift-master-controllers

## Wait for things to initalize
sleep 120

## grep logs for the RBAC errors
journalctl -u atomic-openshift-master-api|grep -E 'named "myimage[0-9]+" in namespace "testimages"'

## grep docker registry logs entries which contain the testimages namespace
oc logs $(oc get pods -n default --config /etc/origin/master/admin.kubeconfig|grep docker-registry|grep Running|awk '{print $1}') -n default --config /etc/origin/master/admin.kubeconfig|grep testimages

Actual results:
Thousands of log entries similar to this:
atomic-openshift-master-api[1878]: I0109 08:53:25.734369    1878 rbac.go:116] RBAC DENY: user "system:anonymous" groups ["system:unauthenticated"] cannot "get" resource "imagestreams/layers.image.openshift.io" named "testimage" in namespace "openshift"

Expected results:
At loglevel 2, Customer does not expect to see these types of results.  Lowering the loglevel is not advisable as it removes more valid log entries.  Customer is having difficulty filtering this information out.

Additional info:
Tested in QuickLab OpenShift 3.6 environment and was not able to replicate issue, I then upgraded that environment to 3.7 and repeated.  I was able to replicate as reported.

Comment 1 Greg Rodriguez II 2018-02-08 22:53:10 UTC
Is a route cause known for this issue and is there any cause for concern with these messages or is this just logspam?

Comment 3 Ben Parees 2018-02-14 15:43:14 UTC
Michal what makes you suspect the registry is making these anonymous requests?

(And Oleg, assuming it is the registry, do you have any theory about what's causing it?)

Comment 5 Oleg Bulatov 2018-02-15 10:18:15 UTC
https://github.com/openshift/origin/issues/18564

Comment 6 Michal Fojtik 2018-02-15 12:55:21 UTC
Yeah, it is either signature importer or the docker plugin that fetches signatures from registry.

Comment 7 Ben Parees 2018-02-15 13:51:17 UTC
Michal did you ever land your PRs to fix this stuff? (ie allow us to disable the import controller and/or stop it looking up signatures against anything except the redhat registry?)  If not, when you do you plan to do so?

Comment 8 Ben Parees 2018-02-23 01:05:41 UTC
Oleg, please talk to Michal Fojtik about this in the office and find out where his PRs are/what state they are in and how they can be moved forward to resolve this.

Comment 9 Oleg Bulatov 2018-02-23 08:52:29 UTC
I talked to him a few day ago, here is his PR: https://github.com/containers/image/pull/383

But, as I told on scrum, I don't understand why we try to import signatures for managed (pushed into the registry) images. We use a magic library that can get signatures for an image not only from the registry, but from local files too, but do we need to import signatures for managed images from local files?

I cannot find non-marketing description for image signatures, so I don't know where to look for answers.

Comment 10 Oleg Bulatov 2018-02-23 14:17:04 UTC
https://github.com/openshift/origin/pull/18733

Comment 12 Dongbo Yan 2018-03-06 09:55:14 UTC
Hi, Oleg Bulatov

I follow the reproduce steps in description, but cannot get the error.
Could you please help provide some simple steps to reproduce this issue? thanks in advance

Comment 13 Dongbo Yan 2018-03-06 10:05:55 UTC
Test with
# openshift version
openshift v3.7.36
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8

Comment 14 Oleg Bulatov 2018-03-06 11:07:32 UTC
1. run OpenShift server with --loglevel=5,
2. secure registry,
3. push an image to the registry.

You may also want to edit dockerImageReference to use router, and somehow get a trusted certificate, otherwise you'll get

> Failed to get signatures for {{{registry.127.0.0.1.nip.io myproject/busybox-glibc} sha256:eb31109cc48cc7df6d77e559fbb221ef0e1497534538f7ce88f8306ee72b295a}} due to: pinging docker registry returned: Get https://registry.127.0.0.1.nip.io/v2/: x509: certificate signed by unknown authority

Comment 15 Dongbo Yan 2018-03-07 04:46:53 UTC
Thanks for your help, could reproduce this issue with v3.7.36, get error like below:
time="2018-03-07T04:37:16.900931775Z" level=error msg="OpenShift access denied: User \"system:anonymous\" cannot get imagestreams/layers.image.openshift.io in project \"install-test\"" go.version=go1.8.3 http.request.host="docker-registry.default.svc:5000" http.request.id=bd3014e5-030f-4b49-85ed-de0b7f5367a9 http.request.method=GET http.request.remoteaddr="10.128.0.1:53524" http.request.uri="/extensions/v2/install-test/nodejs-mongodb-example/signatures/sha256:b36340c92bf04e3b0bafbf893bf5a0aae2b38f6760e4ab9237c9e45f760a61f7" http.request.useragent="Go-http-client/1.1" instance.id=5f45af42-c13f-46ba-baeb-96d4e8d9e2df openshift.auth.user=anonymous openshift.logger=registry vars.digest="sha256:b36340c92bf04e3b0bafbf893bf5a0aae2b38f6760e4ab9237c9e45f760a61f7" vars.name="install-test/nodejs-mongodb-example" 

Verified with
openshift v3.9.3
kubernetes v1.9.1+a0ce1bc657
etcd 3.2.16

No similar error in docker-registry pod log

Comment 19 errata-xmlrpc 2018-03-28 14:26:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489


Note You need to log in before you can comment on or make changes to this bug.