As this is becoming important, I raised the Customer Escalation Flag.

I think we have a working prototype that should meet offline use cases.  I need to verify our pass rate in that configuration first before going further.

Restoring assignee to Clayton, as it stood after comment 10.

Waiting for https://bugzilla.redhat.com/show_bug.cgi?id=1843232 to be fixed so that the updated e2e tests pass.  That gets the core suite to passing.  Next steps are then to verify the extended suites still pass, to add a detection mechanism that ensures the tests don't regress (they don't start referencing images that aren't mirrored) and documentation of the process in the CI tooling.

Comment 17 talks about bug 1843232, which was closed as a dup of bug 1844596, which is VERIFIED.  So probably unblocked there.  This bug is still assigned to Clayton, but doesn't seem like a blocker for 4.6.  Punting to 4.7.  Also, what happened to "every bug every sprint"?  I don't see any UpcomingSprint keywording here...

*** Bug 1898669 has been marked as a duplicate of this bug. ***

I believe this was assigned to me in error.  Can the owner take me off the notification/watchlist?

The linked origin#24887 is your PR.  Was someone else lined up to carry the PR for you?  Seems like both you and Gabe have been active on the PR recently.

I see Clayton Coleman is working in that record, but I am not.  Because our names our similar, things get mixed up from time to time.  I do not work on this.

Oops, silly auto-complete, and sloppy me.  Sorry for pulling you in, Clarence.

You should not need to generate an ICSP.  You have to run "openshift-tests run X --from-repository=YOUR_OFFLINE_REPO" while disconnected.

I've changed the description of the bug to address the key part (which is fixed - that we use a consistent set of images.  If you can open follow on bugs for the other issues you found and assign to me we can fix those separately, while the core fix has been verified in the CI infrastructure.

Thanks, Clayton, retest it, steps:

1, Compile the `openshift-tests` binary from the master branch. Check this new sub-command `images` help info.
[root@preserve-olm-env origin]# make build
go build -mod=vendor -trimpath -ldflags "-s -w -X github.com/openshift/origin/pkg/version.versionFromGit="v4.1.0-3600-gaf3f04c" -X github.com/openshift/origin/pkg/version.commitFromGit="af3f04c" -X github.com/openshift/origin/pkg/version.gitTreeState="clean" -X github.com/openshift/origin/pkg/version.buildDate="2021-02-09T09:33:49Z" " github.com/openshift/origin/cmd/openshift-tests

[root@preserve-olm-env origin]# ls -l openshift-tests 
-rwxr-xr-x. 1 root root 161831136 Feb  9 09:35 openshift-tests
[root@preserve-olm-env origin]# ./openshift-tests images --help
Creates a mapping to mirror test images to a private registry
...

2, Login in a disconnected cluster.
[root@preserve-olm-env origin]# export KUBECONFIG=/data/bug-kubeconfig
[root@preserve-olm-env origin]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2021-02-09-024347   True        False         74m     Cluster version is 4.7.0-0.nightly-2021-02-09-024347

3, Mirror the image to the registry that connected to this disconnected cluster.
[root@preserve-olm-env origin]# ./openshift-tests images --to-repository jiazha0210.mirror-registry.qe.azure.devcluster.openshift.com:5000/qe/community-e2e-images > mapping 
[root@preserve-olm-env origin]# oc image mirror -f mapping -a /data/mirror_docker.conf 
W0210 02:56:13.736667    6012 manifest.go:440] Chose linux/amd64 manifest from the manifest list.
W0210 02:56:13.924501    6012 manifest.go:440] Chose linux/amd64 manifest from the manifest list.
W0210 02:56:14.135797    6012 manifest.go:440] Chose linux/amd64 manifest from the manifest list.
...
jiazha0210.mirror-registry.qe.azure.devcluster.openshift.com:5000/
  qe/community-e2e-images
    blobs:
      quay.io/openshift/community-e2e-images sha256:36c0847f9cd6774286a01e72011a173a77d007945069ff8257fc977b0ac2965f
...
mmunity-e2e-images:e2e-23-docker-io-library-nginx-1-14-alpine-yxm61cIl0fz2I0Zz
sha256:e30919918299988b318f0208e7fd264dee21a6be9d74bbd9f7fc15e78eade9b4 jiazha0210.mirror-registry.qe.azure.devcluster.openshift.com:5000/qe/community-e2e-images:e2e-11-k8s-gcr-io-build-image-debian-iptables-buster-v1-3-0-2IwrNjgvp777TKmc
I0210 03:00:17.648354    6012 manifest.go:499] warning: Digests are not preserved with schema version 1 images. Support for schema version 1 images will be removed in a future release
sha256:a2d0a2a47137e63e102168e62c02d430bc24135a2f733891a056898856e91281 jiazha0210.mirror-registry.qe.azure.devcluster.openshift.com:5000/qe/community-e2e-images:e2e-registry-svc-ci-openshift-org-ocp-4-7-test-build-roots2i-ZzDWhWn0wPB9cLFM
info: Mirroring completed in 3m47.94s (20.72MB/s)


4, Run some test cases without specifying the mirrored images. Failed as expected due to pulling image error.
[root@preserve-olm-env origin]# ./openshift-tests run all --dry-run|grep "[sig-network] multicast"|./openshift-tests run -f -
openshift-tests version: v4.1.0-3600-gaf3f04c
started: (0/1/3) "[sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-multitenant, redhat/openshift-ovs-networkpolicy' should block multicast traffic in namespaces where it is disabled [Suite:openshift/conformance/parallel]"

started: (0/2/3) "[sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-subnet' should block multicast traffic [Suite:openshift/conformance/parallel]"

started: (0/3/3) "[sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-multitenant, redhat/openshift-ovs-networkpolicy' should allow multicast traffic in namespaces where it is enabled [Suite:openshift/conformance/parallel]"
skip [github.com/openshift/origin/test/extended/networking/util.go:338]: Not using one of the specified plugins

skipped: (4.5s) 2021-02-10T06:19:02 "[sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-subnet' should block multicast traffic [Suite:openshift/conformance/parallel]"
...
[AfterEach] [sig-network] multicast
  github.com/openshift/origin/test/extended/util/client.go:139
STEP: Destroying namespace "e2e-test-multicast-rcl5t" for this suite.
fail [github.com/openshift/origin/test/extended/networking/multicast.go:128]: Unexpected error:
    <*errors.errorString | 0xc00115ad90>: {
        s: "Gave up after waiting 5m0s for pod \"multicast-1\" to be \"running\"",
    }
    Gave up after waiting 5m0s for pod "multicast-1" to be "running"
occurred
...
Feb 10 06:23:13.419 I ns/e2e-test-multicast-25448 pod/multicast-3 node/jiazha0210-02100043-worker-northcentralus-2 reason/BackOff Back-off pulling image "quay.io/openshift/community-e2e-images:e2e-docker-io-openshift-test-multicast-latest-4AxcBBxKg_prX34z" (4 times)


Failing tests:

[sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-multitenant, redhat/openshift-ovs-networkpolicy' should allow multicast traffic in namespaces where it is enabled [Suite:openshift/conformance/parallel]
[sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-multitenant, redhat/openshift-ovs-networkpolicy' should block multicast traffic in namespaces where it is disabled [Suite:openshift/conformance/parallel]
error: 2 fail, 0 pass, 1 skip (5m7s)

5, Run some test cases without specifying the mirrored images. 
[root@preserve-olm-env origin]# ./openshift-tests run all --dry-run|grep "[sig-network] multicast"|./openshift-tests run --from-repository jiazha0210.mirror-registry.qe.azure.devcluster.openshift.com:5000/qe/community-e2e-images -f -
openshift-tests version: v4.1.0-3600-gaf3f04c
started: (0/1/3) "[sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-multitenant, redhat/openshift-ovs-networkpolicy' should allow multicast traffic in namespaces where it is enabled [Suite:openshift/conformance/parallel]"

started: (0/2/3) "[sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-subnet' should block multicast traffic [Suite:openshift/conformance/parallel]"

started: (0/3/3) "[sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-multitenant, redhat/openshift-ovs-networkpolicy' should block multicast traffic in namespaces where it is disabled [Suite:openshift/conformance/parallel]"

skip [github.com/openshift/origin/test/extended/networking/util.go:338]: Not using one of the specified plugins

skipped: (3.3s) 2021-02-10T06:27:39 "[sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-subnet' should block multicast traffic [Suite:openshift/conformance/parallel]"

passed: (28.3s) 2021-02-10T06:28:03 "[sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-multitenant, redhat/openshift-ovs-networkpolicy' should block multicast traffic in namespaces where it is disabled [Suite:openshift/conformance/parallel]"

passed: (28.8s) 2021-02-10T06:28:04 "[sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-multitenant, redhat/openshift-ovs-networkpolicy' should allow multicast traffic in namespaces where it is enabled [Suite:openshift/conformance/parallel]"
...

2 pass, 1 skip (28.8s)

They succeed as expected. LGTM, verify it.

For that panic issue, I report a bug here: https://bugzilla.redhat.com/show_bug.cgi?id=1927131

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633