Bug 1918376 - Image registry pullthrough does not support ICSP, mirroring e2es do not pass
Summary: Image registry pullthrough does not support ICSP, mirroring e2es do not pass
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.7
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.8.0
Assignee: Ricardo Maraschini
QA Contact: XiuJuan Wang
Depends On:
Blocks: 1931564
TreeView+ depends on / blocked
Reported: 2021-01-20 15:14 UTC by Clayton Coleman
Modified: 2022-10-12 03:22 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Image registry was ignoring cluster wide ICSP rules. Consequence: During pull through of images mirrors were ignored causing pull failures in disconnected cluster. Fix: Patched the registry to start pulling through from mirrors if ICSP rules exist for the target repository. Result: Pulling through image from configured mirrors does not fail anymore.
Clone Of:
Last Closed: 2021-07-27 22:36:15 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift cluster-image-registry-operator pull 668 0 None closed Bug 1918376: Allowing system:registry to list ICSP rules 2021-06-28 21:53:38 UTC
Github openshift image-registry pull 277 0 None closed Bug 1918376: ICSP support during Image Registry pullthrough 2021-06-28 21:53:37 UTC
Github openshift image-registry pull 278 0 None closed Bug 1918376: Bumping openshift/library-go 2021-06-28 21:53:34 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 22:36:33 UTC

Description Clayton Coleman 2021-01-20 15:14:42 UTC
As a consequence of feature gap (ICSP not fully implemented across the system for disconnected clusters, https://issues.redhat.com/browse/IR-34) the mirroring e2e tests which verify disconnected clusters cannot pass because the e2e tests use image streams with pullthrough (starting 12/08/2020 with the offline e2e changes to mitigate docker and allow offline e2e testing).  This means roughly 50 e2e tests fail, leaving a gap in our coverage that means we may regress customers without being aware of it in a new release.

Because this is tied to work that requires a feature, a one time deferral from 4.8 may be appropriate but leaving disconnected clusters ungated is also a significant product level risk.  If this bug is deferred, we should open a 4.7.z bug and explicitly defer it, then backport the ICSP behavior if the fix comes within a reasonable time.  Also, please keep environment up to date (since the mirroring jobs are 25% red because of this)

Failure list: https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-mirrors-4.7/1351622431432773632

Comment 2 Clayton Coleman 2021-02-03 19:03:54 UTC
"A one time deferral from 4.7".  Right now we can't even run proxy-e2e tests so we don't even know if any other tests have regressed.

Comment 4 Adam Kaplan 2021-02-22 16:56:50 UTC
Per comments in this issue, opened backport to 4.7.z.

Comment 15 XiuJuan Wang 2021-06-15 02:24:26 UTC
Could pull image whose imagestream enabled pullthough via imagecontentsourcepolicy on restricted cluster.

Step 1: Create a mirror registry without auth. and mirror jenkins image to mirror registry

Pull jenkins image from mirror registry without auth
$ podman pull wxj-611icsp14.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp/release@sha256:c8b838b043b7a13f69749ebd99dcf6982b405d6421ee774c5f41cc7891bf8561 --tls-verify=false
Trying to pull wxj-611icsp14.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp/release@sha256:c8b838b043b7a13f69749ebd99dcf6982b405d6421ee774c5f41cc7891bf8561...Getting image source signatures
Copying blob 33db084abe90 done
Copying blob 99920319b5be done
Copying blob 45d8de60ce97 done
Copying blob 3cbc59484248 done
Copying blob d0c9851d609d done
Copying blob a46440ee71ba done
Copying config 1892869616 done
Writing manifest to image destination
Storing signatures

Check if jenkins imagestream imports successfully with pullthough

$ oc get is jenkins -n openshift -o json | jq -r '.status.tags[1]'
  "items": [
      "created": "2021-06-14T11:41:15Z",
      "dockerImageReference": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8b838b043b7a13f69749ebd99dcf6982b405d6421ee774c5f41cc7891bf8561",
      "generation": 4,
      "image": "sha256:c8b838b043b7a13f69749ebd99dcf6982b405d6421ee774c5f41cc7891bf8561"
  "tag": "latest"

$oc get imagecontentsourcepolicy         image-policy-0 -o json  | jq -r '.spec'
  "repositoryDigestMirrors": [
      "mirrors": [
      "source": "quay.io/openshift-release-dev/ocp-v4.0-art-dev"

Create jenkins application
$oc new-app jenkins-persistent
jenkins pod is running.
$ oc get pods 
NAME               READY   STATUS      RESTARTS   AGE
jenkins-1-5pcll    1/1     Running     0          12h
jenkins-1-deploy   0/1     Completed   0          12h

Image could be pulled from mirror registry via iscp.

Comment 18 errata-xmlrpc 2021-07-27 22:36:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.