Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1918376

Summary: Image registry pullthrough does not support ICSP, mirroring e2es do not pass
Product: OpenShift Container Platform Reporter: Clayton Coleman <ccoleman>
Component: Image RegistryAssignee: Ricardo Maraschini <rmarasch>
Status: CLOSED ERRATA QA Contact: XiuJuan Wang <xiuwang>
Severity: high Docs Contact:
Priority: high    
Version: 4.7CC: adam.kaplan, aos-bugs, obulatov, rmarasch, scuppett, wewang, wking
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Image registry was ignoring cluster wide ICSP rules. Consequence: During pull through of images mirrors were ignored causing pull failures in disconnected cluster. Fix: Patched the registry to start pulling through from mirrors if ICSP rules exist for the target repository. Result: Pulling through image from configured mirrors does not fail anymore.
 Story Points: ---
Clone Of: Environment:
job=release-openshift-ocp-installer-e2e-aws-mirrors-4.7=all
Last Closed: 2021-07-27 22:36:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1931564    

Description Clayton Coleman 2021-01-20 15:14:42 UTC 
As a consequence of feature gap (ICSP not fully implemented across the system for disconnected clusters, https://issues.redhat.com/browse/IR-34) the mirroring e2e tests which verify disconnected clusters cannot pass because the e2e tests use image streams with pullthrough (starting 12/08/2020 with the offline e2e changes to mitigate docker and allow offline e2e testing).  This means roughly 50 e2e tests fail, leaving a gap in our coverage that means we may regress customers without being aware of it in a new release.

Because this is tied to work that requires a feature, a one time deferral from 4.8 may be appropriate but leaving disconnected clusters ungated is also a significant product level risk.  If this bug is deferred, we should open a 4.7.z bug and explicitly defer it, then backport the ICSP behavior if the fix comes within a reasonable time.  Also, please keep environment up to date (since the mirroring jobs are 25% red because of this)

Failure list: https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-mirrors-4.7/1351622431432773632
Comment 2 Clayton Coleman 2021-02-03 19:03:54 UTC 
"A one time deferral from 4.7".  Right now we can't even run proxy-e2e tests so we don't even know if any other tests have regressed.
Comment 4 Adam Kaplan 2021-02-22 16:56:50 UTC 
Per comments in this issue, opened backport to 4.7.z.
Comment 15 XiuJuan Wang 2021-06-15 02:24:26 UTC 
Could pull image whose imagestream enabled pullthough via imagecontentsourcepolicy on restricted cluster.

Step 1: Create a mirror registry without auth. and mirror jenkins image to mirror registry

Pull jenkins image from mirror registry without auth
$ podman pull wxj-611icsp14.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp/release@sha256:c8b838b043b7a13f69749ebd99dcf6982b405d6421ee774c5f41cc7891bf8561 --tls-verify=false
Trying to pull wxj-611icsp14.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp/release@sha256:c8b838b043b7a13f69749ebd99dcf6982b405d6421ee774c5f41cc7891bf8561...Getting image source signatures
Copying blob 33db084abe90 done
Copying blob 99920319b5be done
Copying blob 45d8de60ce97 done
Copying blob 3cbc59484248 done
Copying blob d0c9851d609d done
Copying blob a46440ee71ba done
Copying config 1892869616 done
Writing manifest to image destination
Storing signatures
189286961671ac5015e64c17c3b0e633abe2734a46a39f8f756c37a0d7a9202e

Check if jenkins imagestream imports successfully with pullthough

$ oc get is jenkins -n openshift -o json | jq -r '.status.tags[1]'
{
  "items": [
    {
      "created": "2021-06-14T11:41:15Z",
      "dockerImageReference": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8b838b043b7a13f69749ebd99dcf6982b405d6421ee774c5f41cc7891bf8561",
      "generation": 4,
      "image": "sha256:c8b838b043b7a13f69749ebd99dcf6982b405d6421ee774c5f41cc7891bf8561"
    }
  ],
  "tag": "latest"
}

$oc get imagecontentsourcepolicy         image-policy-0 -o json  | jq -r '.spec'
{
  "repositoryDigestMirrors": [
    {
      "mirrors": [
        "wxj-611icsp14.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp/release"
      ],
      "source": "quay.io/openshift-release-dev/ocp-v4.0-art-dev"
    }
  ]
}

Create jenkins application
$oc new-app jenkins-persistent
 
jenkins pod is running.
$ oc get pods 
NAME               READY   STATUS      RESTARTS   AGE
jenkins-1-5pcll    1/1     Running     0          12h
jenkins-1-deploy   0/1     Completed   0          12h

Image could be pulled from mirror registry via iscp.
Comment 18 errata-xmlrpc 2021-07-27 22:36:15 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438