Bug 2061785 - Image registry uses ICSPs only when source exactly matches image
Summary: Image registry uses ICSPs only when source exactly matches image
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.8
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.10.z
Assignee: Oleg Bulatov
QA Contact: XiuJuan Wang
URL:
Whiteboard:
Depends On: 2014240
Blocks: 2086864
TreeView+ depends on / blocked
 
Reported: 2022-03-08 14:43 UTC by Oleg Bulatov
Modified: 2022-05-16 17:16 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 2014240
Environment:
Last Closed: 2022-03-28 12:03:50 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift image-registry pull 318 0 None open Bug 2061785: Fix ICSP for subrepositories 2022-03-08 14:45:53 UTC
Red Hat Product Errata RHBA-2022:1026 0 None None None 2022-03-28 12:04:12 UTC

Comment 1 XiuJuan Wang 2022-03-09 11:30:39 UTC
Verified on 4.10.0-0.ci.test-2022-03-09-075704-ci-ln-1xv7lx2-latest

spec:
  repositoryDigestMirrors:
  - mirrors:
    - ec2-3-145-100-29.us-east-2.compute.amazonaws.com:5000/openshifttest
    source: quay.io/openshifttest
  - mirrors:
    - ec2-3-145-100-29.us-east-2.compute.amazonaws.com:5000
    source: registry.redhat.io

Senario 1: 
sources and mirrors only defined registry hostname
oc tag registry.redhat.io/rhel8/mysql-80@sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545 mysql:latest --reference-policy=local
$oc create deployment mysql-deployment --image=image-registry.openshift-image-registry.svc:5000/default/mysql:latest -- sleep 300
$oc get pods 

Senario 2: 
sources and mirrors only defined registry hostname and namespace
$oc tag quay.io/openshifttest/skopeo@sha256:426196e376cf045012289d53fec986554241496ed7f38e347fc56505aa8ad322 skopeo:latest --reference-policy=local
$oc create deployment test-deployment --image=image-registry.openshift-image-registry.svc:5000/default/skopeo:latest -- sleep 300
oc get pods


spec:
  repositoryDigestMirrors:
  - mirrors:
    - ec2-3-145-100-29.us-east-2.compute.amazonaws.com:5000/app
    source: registry1-wxj.apps.wxj225a.qe.devcluster.openshift.com/foo/bar/myapp
  - mirrors:
    - ec2-3-145-100-29.us-east-2.compute.amazonaws.com:5000/app
    source: registry2-wxj.apps.wxj225a.qe.devcluster.openshift.com/foo/bar
  - mirrors:
    - ec2-3-145-100-29.us-east-2.compute.amazonaws.com:5000
    source: registry2-wxj.apps.wxj225a.qe.devcluster.openshift.com/foo/bar
  - mirrors:
    - ec2-3-145-100-29.us-east-2.compute.amazonaws.com:5000/registry-example-com
    source: registry4-wxj.apps.wxj225a.qe.devcluster.openshift.com
  - mirrors:
    - ec2-3-145-100-29.us-east-2.compute.amazonaws.com:5000/mirrored-foo
    source: registry5-wxj.apps.wxj225a.qe.devcluster.openshift.com/foo

$for i in {1..5}; do oc run pod$i --image=registry$i-wxj.apps.wxj225a.qe.devcluster.openshift.com/foo/bar/myapp@sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8 -- sleep 300 ; done 

Since the 5 source repositories don't exist, so failed to pull image, but the pull path are correct.

$for i in {1..5}; do oc describe pods pod$i | grep "Failed to pull image"; done
  Warning  Failed          7m55s (x4 over 9m26s)   kubelet            Failed to pull image "registry1-wxj.apps.wxj225a.qe.devcluster.openshift.com/foo/bar/myapp@sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8": rpc error: code = Unknown desc = (Mirrors also failed: [ec2-3-145-100-29.us-east-2.compute.amazonaws.com:5000/app@sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8: reading manifest sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8 in ec2-3-145-100-29.us-east-2.compute.amazonaws.com:5000/app: manifest unknown: manifest unknown]): registry1-wxj.apps.wxj225a.qe.devcluster.openshift.com/foo/bar/myapp@sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8: pinging container registry registry1-wxj.apps.wxj225a.qe.devcluster.openshift.com: Get "https://registry1-wxj.apps.wxj225a.qe.devcluster.openshift.com/v2/": Forbidden
  Warning  Failed          7m4s (x2 over 7m16s)   kubelet            Failed to pull image "registry2-wxj.apps.wxj225a.qe.devcluster.openshift.com/foo/bar/myapp@sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8": rpc error: code = Unknown desc = (Mirrors also failed: [ec2-3-145-100-29.us-east-2.compute.amazonaws.com:5000/myapp@sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8: reading manifest sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8 in ec2-3-145-100-29.us-east-2.compute.amazonaws.com:5000/myapp: manifest unknown: manifest unknown]): registry2-wxj.apps.wxj225a.qe.devcluster.openshift.com/foo/bar/myapp@sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8: pinging container registry registry2-wxj.apps.wxj225a.qe.devcluster.openshift.com: Get "https://registry2-wxj.apps.wxj225a.qe.devcluster.openshift.com/v2/": Forbidden
  Warning  Failed          5m57s (x2 over 6m39s)  kubelet            Failed to pull image "registry2-wxj.apps.wxj225a.qe.devcluster.openshift.com/foo/bar/myapp@sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8": rpc error: code = Unknown desc = (Mirrors also failed: [ec2-3-145-100-29.us-east-2.compute.amazonaws.com:5000/myapp@sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8: reading manifest sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8 in ec2-3-145-100-29.us-east-2.compute.amazonaws.com:5000/myapp: manifest unknown: manifest unknown]
  Warning  Failed          4m32s (x4 over 6m4s)  kubelet            Failed to pull image "registry3-wxj.apps.wxj225a.qe.devcluster.openshift.com/foo/bar/myapp@sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8": rpc error: code = Unknown desc = pinging container registry registry3-wxj.apps.wxj225a.qe.devcluster.openshift.com: Get "https://registry3-wxj.apps.wxj225a.qe.devcluster.openshift.com/v2/": Forbidden
  Warning  Failed          4m10s (x4 over 5m39s)  kubelet            Failed to pull image "registry4-wxj.apps.wxj225a.qe.devcluster.openshift.com/foo/bar/myapp@sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8": rpc error: code = Unknown desc = (Mirrors also failed: [ec2-3-145-100-29.us-east-2.compute.amazonaws.com:5000/registry-example-com/foo/bar/myapp@sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8: reading manifest sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8 in ec2-3-145-100-29.us-east-2.compute.amazonaws.com:5000/registry-example-com/foo/bar/myapp: manifest unknown: manifest unknown]): registry4-wxj.apps.wxj225a.qe.devcluster.openshift.com/foo/bar/myapp@sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8: pinging container registry registry4-wxj.apps.wxj225a.qe.devcluster.openshift.com: Get "https://registry4-wxj.apps.wxj225a.qe.devcluster.openshift.com/v2/": Forbidden
  Warning  Failed          3m35s (x4 over 4m58s)  kubelet            Failed to pull image "registry5-wxj.apps.wxj225a.qe.devcluster.openshift.com/foo/bar/myapp@sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8": rpc error: code = Unknown desc = (Mirrors also failed: [ec2-3-145-100-29.us-east-2.compute.amazonaws.com:5000/mirrored-foo/bar/myapp@sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8: reading manifest sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8 in ec2-3-145-100-29.us-east-2.compute.amazonaws.com:5000/mirrored-foo/bar/myapp: manifest unknown: manifest unknown]): registry5-wxj.apps.wxj225a.qe.devcluster.openshift.com/foo/bar/myapp@sha256:ef3e5d3241c55c6e9e85771e604ee06170755b02bd8c6f8b372f8ec77c4684f8: pinging container registry registry5-wxj.apps.wxj225a.qe.devcluster.openshift.com: Get "https://registry5-wxj.apps.wxj225a.qe.devcluster.openshift.com/v2/": Forbidden

Comment 6 errata-xmlrpc 2022-03-28 12:03:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.10.6 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1026


Note You need to log in before you can comment on or make changes to this bug.