Bug 1976271 - Mirror for ImageContentSourcePolicy does not work it source repo is blocked
Summary: Mirror for ImageContentSourcePolicy does not work it source repo is blocked
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.6.z
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: ---
: 4.7.z
Assignee: Peter Hunt
QA Contact: Weinan Liu
URL:
Whiteboard:
Depends On: 1975976
Blocks: 1976272
TreeView+ depends on / blocked
 
Reported: 2021-06-25 16:23 UTC by OpenShift BugZilla Robot
Modified: 2021-08-03 17:56 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-08-03 17:56:24 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github cri-o cri-o pull 5034 0 None closed [1.20] pull: do check for blocked registries 2021-07-23 20:01:50 UTC
Red Hat Product Errata RHBA-2021:2903 0 None None None 2021-08-03 17:56:49 UTC

Description OpenShift BugZilla Robot 2021-06-25 16:23:53 UTC
+++ This bug was initially created as a clone of Bug #1975976 +++

Description of problem:

Mirror for ImageContentSourcePolicy does not work it source repo is blocked.

ICSP:
  repositoryDigestMirrors:
  - mirrors:
    - api.ocp46shared.tamlab.brq.redhat.com:5000/ibmcom
    source: docker.io/ibmcom

Image:
spec:
  additionalTrustedCA:
    name: my-registry-ca
  registrySources:
    blockedRegistries:
    - docker.io

Mirrored: docker.io/ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0 into my internal repo.

This is a 2 part problem for OCP 4.6.

Pod creates and we see InspectFailed:

  Warning  InspectFailed   1s    kubelet            Failed to inspect image "docker.io/ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0": rpc error: code = Unknown desc = cannot use "docker.io/ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0" because it's blocked

cri-o debug shows:

Jun 24 18:37:25 worker-0.ocp46shared.tamlab.brq.redhat.com crio[1862]: time="2021-06-24 18:37:25.436656165Z" level=info msg="Checking image status: docker.io/ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0" file="server/image_status.go:30" id=c8700162-ce53-49b3-83da-3aba2ce9006d name=/runtime.v1alpha2.ImageService/ImageStatus
Jun 24 18:37:25 worker-0.ocp46shared.tamlab.brq.redhat.com crio[1862]: time="2021-06-24 18:37:25.436825057Z" level=debug msg="Response error: cannot use \"docker.io/ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0\" because it's blocked" file="go-grpc-middleware/chain.go:25" id=c8700162-ce53-49b3-83da-3aba2ce9006d name=/runtime.v1alpha2.ImageService/ImageStatus

It never even tries to use the mirrored registry because it fails at the inspect. This first issue is fixed in OCP 4.8 / crio 1.21 by the removal of that code here: https://github.com/cri-o/cri-o/commit/22417169dbba10f1be6602b04c158625f149e3e6

Would need to be backported to 4.6/4.7.

Moving to OCP 4.8 to test, on 4.8 fc7:
 
Now the pod creates and it fails on a policy rejection:

  Warning  Failed          14m (x4 over 16m)   kubelet            Failed to pull image "docker.io/ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0": rpc error: code = Unknown desc = Source image rejected: Running image docker://ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0 is rejected by policy.

We actually rewrite now and start using the mirrored registry:

Jun 24 20:08:00 worker-0.ocp48hackathon.tamlab.brq.redhat.com crio[1587]: time="2021-06-24 20:08:00.912323154Z" level=debug msg="GET https://api.ocp46shared.tamlab.brq.redhat.com:5000/v2/" file="docker/docker_client.go:548"

That all seems fine until we call IsRunningImageAllowed which is still looking at docker.io:

Jun 24 20:08:00 worker-0.ocp48hackathon.tamlab.brq.redhat.com crio[1587]: time="2021-06-24 20:08:00.967192527Z" level=debug msg="IsRunningImageAllowed for image docker:docker.io/ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0" file="signature/policy_eval.go:269"

This looks at the policy file and say's nope becuase docker.io is blocked!

Jun 24 20:08:00 worker-0.ocp48hackathon.tamlab.brq.redhat.com crio[1587]: time="2021-06-24 20:08:00.971470154Z" level=debug msg="Requirement 0: denied, done" file="signature/policy_eval.go:280"
Jun 24 20:08:00 worker-0.ocp48hackathon.tamlab.brq.redhat.com crio[1587]: time="2021-06-24 20:08:00.971616614Z" level=debug msg="Error pulling image docker.io/ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0: Source image rejected: Running image docker://ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0 is rejected by policy." file="server/image_pull.go:250" id=4797ef6b-9e87-4b3b-8fb8-94b9960c50b6 name=/runtime.v1alpha2.ImageService/PullImage

This gets enforced here: https://github.com/cri-o/cri-o/blob/release-1.21/vendor/github.com/containers/image/v5/signature/policy_eval.go#L258

Should we expect the policy file to enforce based on the source image or should it not enforce on the mirror image, in the case of image mirroring?

You need to be a cluster admin to setup the ICSP so we should have a higher level of confidence to allow someone to block X but still pull a subset through mirroring.



Version-Release number of selected component (if applicable):
4.6.x+

How reproducible:
Always

Steps to Reproduce:
1. Noted above
2.
3.

Actual results:
Blocked container startup

Expected results:
Mirrored image should work


Additional info:

--- Additional comment from mrobson on 2021-06-24 21:17:07 UTC ---

The same holds true is you use: allowedRegistries and do not specify docker.io as an allowed registry.

The best workaround I can see for this would be to not use blockedRegistries and use allowedRegistries including redhat/quay and specifically docker.io/ibmcom to limit pulls to that single repo.

Would still like to discuss the current behaviour when mirroring is in play though.

--- Additional comment from pehunt on 2021-06-25 16:22:17 UTC ---

I've submitted the attached pr for 4.7. if it's accepted, I'll backport to 4.6 as well

--- Additional comment from pehunt on 2021-06-25 16:23:36 UTC ---

actually, technically this is fixed in 4.8. I'm closing this one and then backporting the bug to the relevant releases

Comment 1 Peter Hunt 2021-06-25 16:25:45 UTC
fixed by attacehd PR

Comment 2 Peter Hunt 2021-07-02 20:39:20 UTC
waiting on upstream merge

Comment 3 Peter Hunt 2021-07-23 20:01:54 UTC
oop PR merged and I missed updating this

Comment 8 errata-xmlrpc 2021-08-03 17:56:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.7.22 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2903


Note You need to log in before you can comment on or make changes to this bug.