Bug 2014240
| Summary: | Image registry uses ICSPs only when source exactly matches image | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Oleg Bulatov <obulatov> | |
| Component: | Image Registry | Assignee: | Oleg Bulatov <obulatov> | |
| Status: | CLOSED ERRATA | QA Contact: | XiuJuan Wang <xiuwang> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 4.8 | CC: | aos-bugs, xiuwang | |
| Target Milestone: | --- | |||
| Target Release: | 4.11.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: |
Cause: the registry used exact match when it applied ICSP
Consequence: pull-through didn't use mirrors for subrepositories
Fix: apply ICSP for subrepositories too
Result: pull-through can use mirrors as it is expected
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 2061785 (view as bug list) | Environment: | ||
| Last Closed: | 2022-08-10 10:38:21 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2061785 | |||
Test on 4.10.0-0.ci.test-2022-02-16-061052-ci-ln-3pb0s5k-latest disconnect cluster
spec:
repositoryDigestMirrors:
- mirrors:
- ec2-3-145-72-98.us-east-2.compute.amazonaws.com:5000/openshifttest
source: quay.io/openshifttest
$oc tag quay.io/openshifttest/skopeo@sha256:426196e376cf045012289d53fec986554241496ed7f38e347fc56505aa8ad322 skopeo:latest --reference-policy=local
$oc create deployment test-deployment --image=image-registry.openshift-image-registry.svc:5000/default/skopeo:latest -- sleep 86400
$oc get pods
NAME READY STATUS RESTARTS AGE
test-deployment-66ffdd5796-8bwl8 1/1 Running 0 2m47s
Hi Oleg, the pull-through doesn't work when icsp only define the registy hostname in source and mirror.
$oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.11.0-0.nightly-2022-02-16-211105 True False 80m Cluster version is 4.11.0-0.nightly-2022-02-16-211105
$oc get imagecontentsourcepolicy image-policy-aosqe -o yaml
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
creationTimestamp: "2022-02-17T08:12:55Z"
generation: 1
name: image-policy-aosqe
resourceVersion: "48561"
uid: 1c7aa830-f9b2-46ac-8216-3df948ce7b38
spec:
repositoryDigestMirrors:
- mirrors:
- ec2-3-144-6-17.us-east-2.compute.amazonaws.com:5000/openshifttest
source: quay.io/openshifttest
- mirrors:
- ec2-3-144-6-17.us-east-2.compute.amazonaws.com:5000
source: registry.redhat.io
- mirrors:
- ec2-3-144-6-17.us-east-2.compute.amazonaws.com:5000
source: registry.stage.redhat.io
- mirrors:
- ec2-3-144-6-17.us-east-2.compute.amazonaws.com:5000/openshift-qe-optional-operators
source: quay.io/openshift-qe-optional-operators
- mirrors:
- ec2-3-144-6-17.us-east-2.compute.amazonaws.com:5000
source: registry-proxy.engineering.redhat.com
$oc describe is mysql
Name: mysql
Namespace: wxj
Created: 35 seconds ago
Labels: <none>
Annotations: openshift.io/image.dockerRepositoryCheck=2022-02-17T09:08:22Z
Image Repository: image-registry.openshift-image-registry.svc:5000/wxj/mysql
Image Lookup: local=false
Unique Images: 1
Tags: 1
latest
tagged from registry.redhat.io/rhel8/mysql-80@sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545
prefer registry pullthrough when referencing this tag
* registry.redhat.io/rhel8/mysql-80@sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545
35 seconds ago
oc set image-lookup mysql
oc create deployment test --image=mysql:latest -- sleep 86400
$oc get pods
NAME READY STATUS RESTARTS AGE
test-6f98679d86-pdh54 0/1 ImagePullBackOff 0 2m55s
Warning Failed 9s kubelet Failed to pull image "image-registry.openshift-image-registry.svc:5000/wxj/mysql@sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545": rpc error: code = Unknown desc = reading manifest sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545 in image-registry.openshift-image-registry.svc:5000/wxj/mysql: unknown: unable to pull manifest from registry.redhat.io/rhel8/mysql-80@sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545: Get "https://registry.redhat.io/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Nice catch! Verified on 4.11.0-0.nightly-2022-02-18-121223 cluster
spec:
repositoryDigestMirrors:
- mirrors:
- ec2-18-188-162-230.us-east-2.compute.amazonaws.com:5000/openshifttest
source: quay.io/openshifttest
- mirrors:
- ec2-18-188-162-230.us-east-2.compute.amazonaws.com:5000
source: registry.redhat.io
- mirrors:
- ec2-18-188-162-230.us-east-2.compute.amazonaws.com:5000
source: registry.stage.redhat.io
- mirrors:
- ec2-18-188-162-230.us-east-2.compute.amazonaws.com:5000/openshift-qe-optional-operators
source: quay.io/openshift-qe-optional-operators
- mirrors:
- ec2-18-188-162-230.us-east-2.compute.amazonaws.com:5000
source: registry-proxy.engineering.redhat.com
Senario 1:
sources and mirrors only defined registry hostname
oc tag registry.redhat.io/rhel8/mysql-80@sha256:144c42e7437f4024eee4ce556d6ead03b4ec7d0ba636a0f1ce0ace9ace49c545 mysql:latest --reference-policy=local
$oc create deployment mysql-deployment --image=image-registry.openshift-image-registry.svc:5000/default/mysql:latest -- sleep 300
$oc get pods
Senario 2:
sources and mirrors only defined registry hostname and namespace
$oc tag quay.io/openshifttest/skopeo@sha256:426196e376cf045012289d53fec986554241496ed7f38e347fc56505aa8ad322 skopeo:latest --reference-policy=local
$oc create deployment test-deployment --image=image-registry.openshift-image-registry.svc:5000/default/skopeo:latest -- sleep 300
oc get pods
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069 |
Description of problem: If ICSP is created for a repository (let's say `registry.redhat.io/openshift4`), this policy be applied not only for this repository, but also for all subrepositories. Version-Release number of selected component (if applicable): 4.8+ How reproducible: Always Steps to Reproduce: 1. create ICSP with `source` like `quay.io` or `registry.redhat.io/openshift4`, let's say mirrors: - quay.io/centos source: fake.example.com/foo 2. import an image fake.example.com/foo/centos (quay.io/centos/centos should be imported) 3. create a pod thats pull this image through the image registry Actual results: The image registry doesn't know how to resolve `fake.example.com/foo/centos` and cannot serve the image. Expected results: The image registry should successfully pull this image from quay.io. Additional info: