Hide Forgot
Description of problem: If ICSP is created for a repository (let's say `registry.redhat.io/openshift4`), this policy be applied not only for this repository, but also for all subrepositories. Version-Release number of selected component (if applicable): 4.8+ How reproducible: Always Steps to Reproduce: 1. create ICSP with `source` like `quay.io` or `registry.redhat.io/openshift4`, let's say mirrors: - quay.io/centos source: fake.example.com/foo 2. import an image fake.example.com/foo/centos (quay.io/centos/centos should be imported) 3. create a pod thats pull this image through the image registry Actual results: The image registry doesn't know how to resolve `fake.example.com/foo/centos` and cannot serve the image. Expected results: The image registry should successfully pull this image from quay.io. Additional info:
Test on 4.10.0-0.ci.test-2022-02-16-061052-ci-ln-3pb0s5k-latest disconnect cluster spec: repositoryDigestMirrors: - mirrors: - ec2-3-145-72-98.us-east-2.compute.amazonaws.com:5000/openshifttest source: quay.io/openshifttest $oc tag quay.io/openshifttest/skopeo@sha256:426196e376cf045012289d53fec986554241496ed7f38e347fc56505aa8ad322 skopeo:latest --reference-policy=local $oc create deployment test-deployment --image=image-registry.openshift-image-registry.svc:5000/default/skopeo:latest -- sleep 86400 $oc get pods NAME READY STATUS RESTARTS AGE test-deployment-66ffdd5796-8bwl8 1/1 Running 0 2m47s
Hi Oleg, the pull-through doesn't work when icsp only define the registy hostname in source and mirror. $oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.0-0.nightly-2022-02-16-211105 True False 80m Cluster version is 4.11.0-0.nightly-2022-02-16-211105 $oc get imagecontentsourcepolicy image-policy-aosqe -o yaml apiVersion: operator.openshift.io/v1alpha1 kind: ImageContentSourcePolicy metadata: creationTimestamp: "2022-02-17T08:12:55Z" generation: 1 name: image-policy-aosqe resourceVersion: "48561" uid: 1c7aa830-f9b2-46ac-8216-3df948ce7b38 spec: repositoryDigestMirrors: - mirrors: - ec2-3-144-6-17.us-east-2.compute.amazonaws.com:5000/openshifttest source: quay.io/openshifttest - mirrors: - ec2-3-144-6-17.us-east-2.compute.amazonaws.com:5000 source: registry.redhat.io - mirrors: - ec2-3-144-6-17.us-east-2.compute.amazonaws.com:5000 source: registry.stage.redhat.io - mirrors: - ec2-3-144-6-17.us-east-2.compute.amazonaws.com:5000/openshift-qe-optional-operators source: quay.io/openshift-qe-optional-operators - mirrors: - ec2-3-144-6-17.us-east-2.compute.amazonaws.com:5000 source: registry-proxy.engineering.redhat.com $oc describe is mysql Name: mysql Namespace: wxj Created: 35 seconds ago Labels: <none> Annotations: openshift.io/image.dockerRepositoryCheck=2022-02-17T09:08:22Z Image Repository: image-registry.openshift-image-registry.svc:5000/wxj/mysql Image Lookup: local=false Unique Images: 1 Tags: 1 latest tagged from registry.redhat.io/rhel8/mysql-80@sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545 prefer registry pullthrough when referencing this tag * registry.redhat.io/rhel8/mysql-80@sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545 35 seconds ago oc set image-lookup mysql oc create deployment test --image=mysql:latest -- sleep 86400 $oc get pods NAME READY STATUS RESTARTS AGE test-6f98679d86-pdh54 0/1 ImagePullBackOff 0 2m55s Warning Failed 9s kubelet Failed to pull image "image-registry.openshift-image-registry.svc:5000/wxj/mysql@sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545": rpc error: code = Unknown desc = reading manifest sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545 in image-registry.openshift-image-registry.svc:5000/wxj/mysql: unknown: unable to pull manifest from registry.redhat.io/rhel8/mysql-80@sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545: Get "https://registry.redhat.io/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Nice catch!
Verified on 4.11.0-0.nightly-2022-02-18-121223 cluster spec: repositoryDigestMirrors: - mirrors: - ec2-18-188-162-230.us-east-2.compute.amazonaws.com:5000/openshifttest source: quay.io/openshifttest - mirrors: - ec2-18-188-162-230.us-east-2.compute.amazonaws.com:5000 source: registry.redhat.io - mirrors: - ec2-18-188-162-230.us-east-2.compute.amazonaws.com:5000 source: registry.stage.redhat.io - mirrors: - ec2-18-188-162-230.us-east-2.compute.amazonaws.com:5000/openshift-qe-optional-operators source: quay.io/openshift-qe-optional-operators - mirrors: - ec2-18-188-162-230.us-east-2.compute.amazonaws.com:5000 source: registry-proxy.engineering.redhat.com Senario 1: sources and mirrors only defined registry hostname oc tag registry.redhat.io/rhel8/mysql-80@sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545 mysql:latest --reference-policy=local $oc create deployment mysql-deployment --image=image-registry.openshift-image-registry.svc:5000/default/mysql:latest -- sleep 300 $oc get pods Senario 2: sources and mirrors only defined registry hostname and namespace $oc tag quay.io/openshifttest/skopeo@sha256:426196e376cf045012289d53fec986554241496ed7f38e347fc56505aa8ad322 skopeo:latest --reference-policy=local $oc create deployment test-deployment --image=image-registry.openshift-image-registry.svc:5000/default/skopeo:latest -- sleep 300 oc get pods
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069