Description of problem: After upgrading from OCP 3.6 to 3.7 we see hundreds of messages within journald like the following (approximately one RBAC DENY error per image which physically resides within the internal Docker registry): atomic-openshift-master-api[1878]: I0109 08:53:25.734369 1878 rbac.go:116] RBAC DENY: user "system:anonymous" groups ["system:unauthenticated"] cannot "get" resource "imagestreams/layers.image.openshift.io" named "testimage" in namespace "openshift" Version-Release number of selected component (if applicable): 3.7 How reproducible: Very, was able to reproduce with QuickLab following steps provided by customer Steps to Reproduce: To make sure we're testing as close to like/like I've provided a list of steps which will hopefully show what we're seeing. Notes: - Please disable atomic-openshift-master-api and atomic-openshift-master-controllers on all nodes but a single instance. This makes troubleshooting easier :) - Perform all remaining operations from the remaining fully operational openshift-master-api node --> Ensure you are running these commands as root unless otherwise specified in the directions ## Check loglevel of master api ps auxwww|grep 'openshift start master api'|grep 'loglevel=2' && echo 'INFO: Loglevel is set' || 'ERROR: Loglevel not set to the correct value' ## Ensure that there aren't any lingering /root/.docker files/dir rm -Rf /root/.docker ## Create docker group if it doesn't exist groupadd docker ## Create local user useradd testuser1 -G docker ## Restart docker daemon systemctl restart docker ## Enable debug mode for your docker registry oc set env dc docker-registry REGISTRY_LOG_LEVEL=debug -n default --config /etc/origin/master/admin.kubeconfig ## Scale the registry to a single replica (makes tracking logs much easier) oc scale dc docker-registry --replicas=1 --config /etc/origin/master/admin.kubeconfig ## Create a new project oc adm new-project testimages --config /etc/origin/master/admin.kubeconfig ## Pull down images docker pull registry.access.redhat.com/rhel7-atomic:7.4-119 docker pull registry.access.redhat.com/rhel7-atomic:7.4-97 docker pull registry.access.redhat.com/rhel7-atomic:7.4-88 docker pull registry.access.redhat.com/rhel7-atomic:7.4-79 docker pull registry.access.redhat.com/rhel7-atomic:7.4-70 docker pull registry.access.redhat.com/rhel7-atomic:7.4-54 docker pull registry.access.redhat.com/rhel7-atomic:7.3-35 docker pull registry.access.redhat.com/rhel7-atomic:7.3-31 docker pull registry.access.redhat.com/rhel7-atomic:7.3-26 docker pull registry.access.redhat.com/rhel7-atomic:7.3-19 docker pull registry.access.redhat.com/rhel7-atomic:7.3-15 docker pull registry.access.redhat.com/rhel7-atomic:7.3-9 docker pull registry.access.redhat.com/rhel7-atomic:7.3-6 export DOCKERREG=$(oc get svc -n default --no-headers --config /etc/origin/master/admin.kubeconfig|grep docker-registry|awk '{print $2}') echo "export DOCKERREG=${DOCKERREG}" >> /home/testuser1/.bash_profile ## Tag Docker images docker tag a9b2247c0747 ${DOCKERREG}:5000/testimages/myimage1:latest docker tag 9d66e4acd0bb ${DOCKERREG}:5000/testimages/myimage2:latest docker tag dfbd36ddc7ca ${DOCKERREG}:5000/testimages/myimage3:latest docker tag 6b5f2e218721 ${DOCKERREG}:5000/testimages/myimage4:latest docker tag 0fec25183884 ${DOCKERREG}:5000/testimages/myimage5:latest docker tag 68a151d90449 ${DOCKERREG}:5000/testimages/myimage6:latest docker tag 1d81b617b801 ${DOCKERREG}:5000/testimages/myimage7:latest docker tag c921f9911a21 ${DOCKERREG}:5000/testimages/myimage8:latest docker tag b2a215996d7a ${DOCKERREG}:5000/testimages/myimage9:latest docker tag 99af91042675 ${DOCKERREG}:5000/testimages/myimage10:latest docker tag 8f0f6a9eb42d ${DOCKERREG}:5000/testimages/myimage11:latest docker tag c8833d5bce74 ${DOCKERREG}:5000/testimages/myimage12:latest docker tag fb7880540f48 ${DOCKERREG}:5000/testimages/myimage13:latest ## Switch users account to your testuser1 sudo -i -u testuser1 ## Login to OpenShift with a user who can push images into our custom namespace 'testimages' oc login -u ....... ## Login to your internal registry docker login -u empty -p $(oc whoami -t) ${DOCKERREG}:5000 ## Push your images into the internal registry docker push ${DOCKERREG}:5000/testimages/myimage1:latest docker push ${DOCKERREG}:5000/testimages/myimage2:latest docker push ${DOCKERREG}:5000/testimages/myimage3:latest docker push ${DOCKERREG}:5000/testimages/myimage4:latest docker push ${DOCKERREG}:5000/testimages/myimage5:latest docker push ${DOCKERREG}:5000/testimages/myimage6:latest docker push ${DOCKERREG}:5000/testimages/myimage7:latest docker push ${DOCKERREG}:5000/testimages/myimage8:latest docker push ${DOCKERREG}:5000/testimages/myimage9:latest docker push ${DOCKERREG}:5000/testimages/myimage10:latest docker push ${DOCKERREG}:5000/testimages/myimage11:latest docker push ${DOCKERREG}:5000/testimages/myimage12:latest docker push ${DOCKERREG}:5000/testimages/myimage13:latest ## Switch back to root exit ## Stop our services systemctl stop atomic-openshift-master-controllers systemctl stop atomic-openshift-master-api ## Wait a minute or so sleep 30 ## Start everything back up again systemctl start atomic-openshift-master-api systemctl start atomic-openshift-master-controllers ## Wait for things to initalize sleep 120 ## grep logs for the RBAC errors journalctl -u atomic-openshift-master-api|grep -E 'named "myimage[0-9]+" in namespace "testimages"' ## grep docker registry logs entries which contain the testimages namespace oc logs $(oc get pods -n default --config /etc/origin/master/admin.kubeconfig|grep docker-registry|grep Running|awk '{print $1}') -n default --config /etc/origin/master/admin.kubeconfig|grep testimages Actual results: Thousands of log entries similar to this: atomic-openshift-master-api[1878]: I0109 08:53:25.734369 1878 rbac.go:116] RBAC DENY: user "system:anonymous" groups ["system:unauthenticated"] cannot "get" resource "imagestreams/layers.image.openshift.io" named "testimage" in namespace "openshift" Expected results: At loglevel 2, Customer does not expect to see these types of results. Lowering the loglevel is not advisable as it removes more valid log entries. Customer is having difficulty filtering this information out. Additional info: Tested in QuickLab OpenShift 3.6 environment and was not able to replicate issue, I then upgraded that environment to 3.7 and repeated. I was able to replicate as reported.
Is a route cause known for this issue and is there any cause for concern with these messages or is this just logspam?
Michal what makes you suspect the registry is making these anonymous requests? (And Oleg, assuming it is the registry, do you have any theory about what's causing it?)
https://github.com/openshift/origin/issues/18564
Yeah, it is either signature importer or the docker plugin that fetches signatures from registry.
Michal did you ever land your PRs to fix this stuff? (ie allow us to disable the import controller and/or stop it looking up signatures against anything except the redhat registry?) If not, when you do you plan to do so?
Oleg, please talk to Michal Fojtik about this in the office and find out where his PRs are/what state they are in and how they can be moved forward to resolve this.
I talked to him a few day ago, here is his PR: https://github.com/containers/image/pull/383 But, as I told on scrum, I don't understand why we try to import signatures for managed (pushed into the registry) images. We use a magic library that can get signatures for an image not only from the registry, but from local files too, but do we need to import signatures for managed images from local files? I cannot find non-marketing description for image signatures, so I don't know where to look for answers.
https://github.com/openshift/origin/pull/18733
Hi, Oleg Bulatov I follow the reproduce steps in description, but cannot get the error. Could you please help provide some simple steps to reproduce this issue? thanks in advance
Test with # openshift version openshift v3.7.36 kubernetes v1.7.6+a08f5eeb62 etcd 3.2.8
1. run OpenShift server with --loglevel=5, 2. secure registry, 3. push an image to the registry. You may also want to edit dockerImageReference to use router, and somehow get a trusted certificate, otherwise you'll get > Failed to get signatures for {{{registry.127.0.0.1.nip.io myproject/busybox-glibc} sha256:eb31109cc48cc7df6d77e559fbb221ef0e1497534538f7ce88f8306ee72b295a}} due to: pinging docker registry returned: Get https://registry.127.0.0.1.nip.io/v2/: x509: certificate signed by unknown authority
Thanks for your help, could reproduce this issue with v3.7.36, get error like below: time="2018-03-07T04:37:16.900931775Z" level=error msg="OpenShift access denied: User \"system:anonymous\" cannot get imagestreams/layers.image.openshift.io in project \"install-test\"" go.version=go1.8.3 http.request.host="docker-registry.default.svc:5000" http.request.id=bd3014e5-030f-4b49-85ed-de0b7f5367a9 http.request.method=GET http.request.remoteaddr="10.128.0.1:53524" http.request.uri="/extensions/v2/install-test/nodejs-mongodb-example/signatures/sha256:b36340c92bf04e3b0bafbf893bf5a0aae2b38f6760e4ab9237c9e45f760a61f7" http.request.useragent="Go-http-client/1.1" instance.id=5f45af42-c13f-46ba-baeb-96d4e8d9e2df openshift.auth.user=anonymous openshift.logger=registry vars.digest="sha256:b36340c92bf04e3b0bafbf893bf5a0aae2b38f6760e4ab9237c9e45f760a61f7" vars.name="install-test/nodejs-mongodb-example" Verified with openshift v3.9.3 kubernetes v1.9.1+a0ce1bc657 etcd 3.2.16 No similar error in docker-registry pod log
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0489