Bug 2014240 - Image registry uses ICSPs only when source exactly matches image
Summary: Image registry uses ICSPs only when source exactly matches image
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.8
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.11.0
Assignee: Oleg Bulatov
QA Contact: XiuJuan Wang
Depends On:
Blocks: 2061785
TreeView+ depends on / blocked
Reported: 2021-10-14 17:15 UTC by Oleg Bulatov
Modified: 2022-08-10 10:38 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: the registry used exact match when it applied ICSP Consequence: pull-through didn't use mirrors for subrepositories Fix: apply ICSP for subrepositories too Result: pull-through can use mirrors as it is expected
Clone Of:
: 2061785 (view as bug list)
Last Closed: 2022-08-10 10:38:21 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift image-registry pull 311 0 None open Bug 2014240: Fix ICSP for subrepositories 2022-02-14 12:57:08 UTC
Github openshift image-registry pull 312 0 None open Bug 2014240: Fix ICSP for whole mirror registries 2022-02-17 11:59:55 UTC
Red Hat Product Errata RHSA-2022:5069 0 None None None 2022-08-10 10:38:47 UTC

Description Oleg Bulatov 2021-10-14 17:15:18 UTC
Description of problem:

If ICSP is created for a repository (let's say `registry.redhat.io/openshift4`), this policy be applied not only for this repository, but also for all subrepositories.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. create ICSP with `source` like `quay.io` or `registry.redhat.io/openshift4`, let's say

    - quay.io/centos
    source: fake.example.com/foo

2. import an image fake.example.com/foo/centos (quay.io/centos/centos should be imported)
3. create a pod thats pull this image through the image registry

Actual results:

The image registry doesn't know how to resolve `fake.example.com/foo/centos` and cannot serve the image.

Expected results:

The image registry should successfully pull this image from quay.io.

Additional info:

Comment 10 XiuJuan Wang 2022-02-16 12:28:07 UTC
Test on 4.10.0-0.ci.test-2022-02-16-061052-ci-ln-3pb0s5k-latest disconnect cluster

  - mirrors:
    - ec2-3-145-72-98.us-east-2.compute.amazonaws.com:5000/openshifttest
    source: quay.io/openshifttest

$oc tag quay.io/openshifttest/skopeo@sha256:426196e376cf045012289d53fec986554241496ed7f38e347fc56505aa8ad322 skopeo:latest --reference-policy=local

$oc create deployment test-deployment --image=image-registry.openshift-image-registry.svc:5000/default/skopeo:latest -- sleep 86400

$oc get pods
NAME                               READY   STATUS             RESTARTS        AGE
test-deployment-66ffdd5796-8bwl8   1/1     Running            0               2m47s

Comment 11 XiuJuan Wang 2022-02-17 09:14:59 UTC
Hi Oleg, the pull-through doesn't work when icsp only define the registy hostname in source and mirror.

$oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-02-16-211105   True        False         80m     Cluster version is 4.11.0-0.nightly-2022-02-16-211105

$oc get imagecontentsourcepolicy  image-policy-aosqe -o yaml
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
  creationTimestamp: "2022-02-17T08:12:55Z"
  generation: 1
  name: image-policy-aosqe
  resourceVersion: "48561"
  uid: 1c7aa830-f9b2-46ac-8216-3df948ce7b38
  - mirrors:
    - ec2-3-144-6-17.us-east-2.compute.amazonaws.com:5000/openshifttest
    source: quay.io/openshifttest
  - mirrors:
    - ec2-3-144-6-17.us-east-2.compute.amazonaws.com:5000
    source: registry.redhat.io
  - mirrors:
    - ec2-3-144-6-17.us-east-2.compute.amazonaws.com:5000
    source: registry.stage.redhat.io
  - mirrors:
    - ec2-3-144-6-17.us-east-2.compute.amazonaws.com:5000/openshift-qe-optional-operators
    source: quay.io/openshift-qe-optional-operators
  - mirrors:
    - ec2-3-144-6-17.us-east-2.compute.amazonaws.com:5000
    source: registry-proxy.engineering.redhat.com

$oc describe is mysql
Name:			mysql
Namespace:		wxj
Created:		35 seconds ago
Labels:			<none>
Annotations:		openshift.io/image.dockerRepositoryCheck=2022-02-17T09:08:22Z
Image Repository:	image-registry.openshift-image-registry.svc:5000/wxj/mysql
Image Lookup:		local=false
Unique Images:		1
Tags:			1

  tagged from registry.redhat.io/rhel8/mysql-80@sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545
    prefer registry pullthrough when referencing this tag

  * registry.redhat.io/rhel8/mysql-80@sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545
      35 seconds ago

oc set image-lookup mysql
oc create deployment test --image=mysql:latest  -- sleep 86400

$oc get pods
 NAME                               READY   STATUS             RESTARTS   AGE
test-6f98679d86-pdh54              0/1     ImagePullBackOff   0          2m55s

  Warning  Failed          9s    kubelet            Failed to pull image "image-registry.openshift-image-registry.svc:5000/wxj/mysql@sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545": rpc error: code = Unknown desc = reading manifest sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545 in image-registry.openshift-image-registry.svc:5000/wxj/mysql: unknown: unable to pull manifest from registry.redhat.io/rhel8/mysql-80@sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545: Get "https://registry.redhat.io/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Comment 12 Oleg Bulatov 2022-02-17 11:59:39 UTC
Nice catch!

Comment 16 XiuJuan Wang 2022-02-21 06:26:34 UTC
Verified on 4.11.0-0.nightly-2022-02-18-121223 cluster

  - mirrors:
    - ec2-18-188-162-230.us-east-2.compute.amazonaws.com:5000/openshifttest
    source: quay.io/openshifttest
  - mirrors:
    - ec2-18-188-162-230.us-east-2.compute.amazonaws.com:5000
    source: registry.redhat.io
  - mirrors:
    - ec2-18-188-162-230.us-east-2.compute.amazonaws.com:5000
    source: registry.stage.redhat.io
  - mirrors:
    - ec2-18-188-162-230.us-east-2.compute.amazonaws.com:5000/openshift-qe-optional-operators
    source: quay.io/openshift-qe-optional-operators
  - mirrors:
    - ec2-18-188-162-230.us-east-2.compute.amazonaws.com:5000
    source: registry-proxy.engineering.redhat.com

Senario 1: 
sources and mirrors only defined registry hostname
oc tag registry.redhat.io/rhel8/mysql-80@sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545 mysql:latest --reference-policy=local
$oc create deployment mysql-deployment --image=image-registry.openshift-image-registry.svc:5000/default/mysql:latest -- sleep 300
$oc get pods 

Senario 2: 
sources and mirrors only defined registry hostname and namespace
$oc tag quay.io/openshifttest/skopeo@sha256:426196e376cf045012289d53fec986554241496ed7f38e347fc56505aa8ad322 skopeo:latest --reference-policy=local
$oc create deployment test-deployment --image=image-registry.openshift-image-registry.svc:5000/default/skopeo:latest -- sleep 300
oc get pods

Comment 20 errata-xmlrpc 2022-08-10 10:38:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.