Bug 1976271

+++ This bug was initially created as a clone of Bug #1975976 +++

Description of problem:

Mirror for ImageContentSourcePolicy does not work it source repo is blocked.

ICSP:
  repositoryDigestMirrors:
  - mirrors:
    - api.ocp46shared.tamlab.brq.redhat.com:5000/ibmcom
    source: docker.io/ibmcom

Image:
spec:
  additionalTrustedCA:
    name: my-registry-ca
  registrySources:
    blockedRegistries:
    - docker.io

Mirrored: docker.io/ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0 into my internal repo.

This is a 2 part problem for OCP 4.6.

Pod creates and we see InspectFailed:

  Warning  InspectFailed   1s    kubelet            Failed to inspect image "docker.io/ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0": rpc error: code = Unknown desc = cannot use "docker.io/ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0" because it's blocked

cri-o debug shows:

Jun 24 18:37:25 worker-0.ocp46shared.tamlab.brq.redhat.com crio[1862]: time="2021-06-24 18:37:25.436656165Z" level=info msg="Checking image status: docker.io/ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0" file="server/image_status.go:30" id=c8700162-ce53-49b3-83da-3aba2ce9006d name=/runtime.v1alpha2.ImageService/ImageStatus
Jun 24 18:37:25 worker-0.ocp46shared.tamlab.brq.redhat.com crio[1862]: time="2021-06-24 18:37:25.436825057Z" level=debug msg="Response error: cannot use \"docker.io/ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0\" because it's blocked" file="go-grpc-middleware/chain.go:25" id=c8700162-ce53-49b3-83da-3aba2ce9006d name=/runtime.v1alpha2.ImageService/ImageStatus

It never even tries to use the mirrored registry because it fails at the inspect. This first issue is fixed in OCP 4.8 / crio 1.21 by the removal of that code here: https://github.com/cri-o/cri-o/commit/22417169dbba10f1be6602b04c158625f149e3e6

Would need to be backported to 4.6/4.7.

Moving to OCP 4.8 to test, on 4.8 fc7:
 
Now the pod creates and it fails on a policy rejection:

  Warning  Failed          14m (x4 over 16m)   kubelet            Failed to pull image "docker.io/ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0": rpc error: code = Unknown desc = Source image rejected: Running image docker://ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0 is rejected by policy.

We actually rewrite now and start using the mirrored registry:

Jun 24 20:08:00 worker-0.ocp48hackathon.tamlab.brq.redhat.com crio[1587]: time="2021-06-24 20:08:00.912323154Z" level=debug msg="GET https://api.ocp46shared.tamlab.brq.redhat.com:5000/v2/" file="docker/docker_client.go:548"

That all seems fine until we call IsRunningImageAllowed which is still looking at docker.io:

Jun 24 20:08:00 worker-0.ocp48hackathon.tamlab.brq.redhat.com crio[1587]: time="2021-06-24 20:08:00.967192527Z" level=debug msg="IsRunningImageAllowed for image docker:docker.io/ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0" file="signature/policy_eval.go:269"

This looks at the policy file and say's nope becuase docker.io is blocked!

Jun 24 20:08:00 worker-0.ocp48hackathon.tamlab.brq.redhat.com crio[1587]: time="2021-06-24 20:08:00.971470154Z" level=debug msg="Requirement 0: denied, done" file="signature/policy_eval.go:280"
Jun 24 20:08:00 worker-0.ocp48hackathon.tamlab.brq.redhat.com crio[1587]: time="2021-06-24 20:08:00.971616614Z" level=debug msg="Error pulling image docker.io/ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0: Source image rejected: Running image docker://ibmcom/ibm-apiconnect-operator-bundle@sha256:548c544ee334bf6da51e7b40aaa5feaf61e9bad010d5e5629f7e922d510e1ce0 is rejected by policy." file="server/image_pull.go:250" id=4797ef6b-9e87-4b3b-8fb8-94b9960c50b6 name=/runtime.v1alpha2.ImageService/PullImage

This gets enforced here: https://github.com/cri-o/cri-o/blob/release-1.21/vendor/github.com/containers/image/v5/signature/policy_eval.go#L258

Should we expect the policy file to enforce based on the source image or should it not enforce on the mirror image, in the case of image mirroring?

You need to be a cluster admin to setup the ICSP so we should have a higher level of confidence to allow someone to block X but still pull a subset through mirroring.



Version-Release number of selected component (if applicable):
4.6.x+

How reproducible:
Always

Steps to Reproduce:
1. Noted above
2.
3.

Actual results:
Blocked container startup

Expected results:
Mirrored image should work


Additional info:

--- Additional comment from mrobson on 2021-06-24 21:17:07 UTC ---

The same holds true is you use: allowedRegistries and do not specify docker.io as an allowed registry.

The best workaround I can see for this would be to not use blockedRegistries and use allowedRegistries including redhat/quay and specifically docker.io/ibmcom to limit pulls to that single repo.

Would still like to discuss the current behaviour when mirroring is in play though.

--- Additional comment from pehunt on 2021-06-25 16:22:17 UTC ---

I've submitted the attached pr for 4.7. if it's accepted, I'll backport to 4.6 as well

--- Additional comment from pehunt on 2021-06-25 16:23:36 UTC ---

actually, technically this is fixed in 4.8. I'm closing this one and then backporting the bug to the relevant releases