Hi Vinu,

Thanks for the information, I was using redhat-operator-index for testing and wasn't able to reproduce. I'll try certified-redhat-index instead. Also, thanks for confirming this issue with docker.io/registry, that makes testing a lot easier. I think I have what I need for now, but will let you know if I have any trouble.

Thanks,

Ross

Hello Ross,

Thank you for your update.

We have uploaded the outputs of the mirroring commands in both certified-opearator-index and redhat-operator-index cases. You can also see the tree structure of the v2 directory that the command creates after the to-file mirroring. Please let me know if you need any more data. And, please note that, as I mentioned earlier, CU is ready for a remote session if you need to check this in their environment.

Thanks,
Vinu K

Ross , Vinu - Spoke with Juha on phone...
He would appreciate it if you can let him know when Ross can reproduce this.
Juha can reproduce it consistently.

Colum Gaynor Senior Customer Success Manager

1, get the `oc` client which contains the fixed PR, as follows.

[cloud-user@preserve-olm-env bug-2049133]$ ./oc version -o yaml
clientVersion:
  buildDate: "2022-02-11T20:25:25Z"
  compiler: gc
  gitCommit: d7b5d2b9763c73b28b1dbf8913b36df4c8ffc3de
  gitTreeState: clean
  gitVersion: 4.11.0-202202111945.p0.gd7b5d2b.assembly.stream-d7b5d2b
  goVersion: go1.17.5
  major: ""
  minor: ""
  platform: linux/amd64
openshiftVersion: 4.11.0-0.nightly-2022-02-12-075213
releaseClientVersion: 4.11.0-0.nightly-2022-02-12-075213
serverVersion:
  buildDate: "2022-02-04T23:38:54Z"
  compiler: gc
  gitCommit: 6f5a5295923a614a4202a7ad274b38b69f9ca8c0
  gitTreeState: clean
  gitVersion: v1.23.3+f14faf2
  goVersion: go1.17.5
  major: "1"
  minor: "23"
  platform: linux/amd64

2, Prune the certified-operator-index to get the `sriov-fec` operator by using `opm`, as follows,
[cloud-user@preserve-olm-env bug-2049133]$ opm index prune -f registry.redhat.io/redhat/certified-operator-index:v4.9 -p sriov-fec -t quay.io/olmqe/sriov-fec:v4.9
WARN[0000] DEPRECATION NOTICE:
Sqlite-based catalogs and their related subcommands are deprecated. Support for
them will be removed in a future release. Please migrate your catalog workflows
to the new file-based catalog format. 
INFO[0000] pruning the index                             packages="[sriov-fec]"
INFO[0000] Pulling previous image registry.redhat.io/redhat/certified-operator-index:v4.9 to get metadata  packages="[sriov-fec]"
INFO[0000] running /usr/bin/podman pull registry.redhat.io/redhat/certified-operator-index:v4.9  packages="[sriov-fec]"
INFO[0002] running /usr/bin/podman pull registry.redhat.io/redhat/certified-operator-index:v4.9  packages="[sriov-fec]"
INFO[0003] Getting label data from previous image        packages="[sriov-fec]"
INFO[0003] running podman inspect                        packages="[sriov-fec]"
INFO[0003] running podman create                         packages="[sriov-fec]"
INFO[0004] running podman cp                             packages="[sriov-fec]"
INFO[0018] running podman rm                             packages="[sriov-fec]"
...
...
INFO[0021] [podman build --format docker -f ./index.Dockerfile301367912 -t quay.io/olmqe/sriov-fec:v4.9 .]  packages="[sriov-fec]"

[cloud-user@preserve-olm-env bug-2049133]$ podman push quay.io/olmqe/sriov-fec:v4.9
...

3, mirror it to local
[cloud-user@preserve-olm-env bug-2049133]$ ./oc adm catalog mirror quay.io/olmqe/sriov-fec:v4.9 file:///local/index 
W0213 20:46:53.914770 2154040 helpers.go:151] Defaulting of registry auth file to "${HOME}/.docker/config.json" is deprecated. The default will be switched to podman config locations in the future version.
...
...
wrote mirroring manifests to manifests-sriov-fec-1644803213

To upload local images to a registry, run:

	oc adm catalog mirror file://local/index/olmqe/sriov-fec:v4.9 REGISTRY/REPOSITORY
deleted dir /tmp/3221597870
...

4, mirror the local file to the registry
[cloud-user@preserve-olm-env bug-2049133]$ ./oc adm catalog mirror file://local/index/olmqe/sriov-fec:v4.9 quay.io/jiazha
W0213 20:47:34.241928 2154053 helpers.go:151] Defaulting of registry auth file to "${HOME}/.docker/config.json" is deprecated. The default will be switched to podman config locations in the future version.
...
sha256:21c1db67da986eecf727ae9c47b5939bbbe0ac6e9a07b2ed23bd2a615280b5e9 quay.io/jiazha/openshift4-ose-sriov-network-device-plugin:55684a3b
error: unable to push manifest to quay.io/jiazha/intel-sriov-fec-operator:36c9eb4d: manifest invalid: manifest invalid
error: unable to push manifest to quay.io/jiazha/intel-sriov-fec-daemon:e7469d15: manifest invalid: manifest invalid
error: unable to push manifest to quay.io/jiazha/intel-n3000-labeler:f86885fc: manifest invalid: manifest invalid
info: Mirroring completed in 1s (0B/s)
error mirroring image: one or more errors occurred
no digest mapping available for file://local/index/olmqe/sriov-fec:v4.9, skip writing to ImageContentSourcePolicy
wrote mirroring manifests to manifests-index/olmqe/sriov-fec-1644803254
deleted dir /tmp/1319547368


As we can see above, after the fix, although I didn't meet the 'missing signature key' error, the three operators' manifests are invalid:

error: unable to push manifest to quay.io/jiazha/intel-sriov-fec-operator:36c9eb4d: manifest invalid: manifest invalid
error: unable to push manifest to quay.io/jiazha/intel-sriov-fec-daemon:e7469d15: manifest invalid: manifest invalid
error: unable to push manifest to quay.io/jiazha/intel-n3000-labeler:f86885fc: manifest invalid: manifest invalid

AFAIK, the quay.io doesn't support the `application/vnd.oci.image.config.v1+json` media type. So, I build a docker registry for another try.

1, Create a docker mirror registry.
[cloud-user@preserve-olm-env bug-2049133]$ docker ps
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
CONTAINER ID  IMAGE                             COMMAND               CREATED         STATUS             PORTS                   NAMES
3b1b80bcd60a  docker.io/library/registry:2.7.0  /etc/docker/regis...  12 minutes ago  Up 12 minutes ago  0.0.0.0:5000->5000/tcp  registry_native_auth

2, mirror local file to it. It works well.
[cloud-user@preserve-olm-env bug-2049133]$ ./oc adm catalog mirror file://local/index/olmqe/sriov-fec:v4.9 localhost:5000/jiazha -a /run/user/1000/containers/auth.json --insecure

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! DEPRECATION NOTICE:
!!   Sqlite-based catalogs are deprecated. Support for them will be removed in a
!!   future release. Please migrate your catalog workflows to the new file-based
!!   catalog format.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

src image has index label for database path: /database/index.db
using index path mapping: /database/index.db:/tmp/1317676454
wrote database to /tmp/1317676454
using database at: /tmp/1317676454/index.db
...
...
sha256:76e63a005dba56abc6e636b98bbf081d763d46f99830c4a5dac6c3c9223b94ab localhost:5000/jiazha/intel-sriov-fec-daemon:e7469d15
uploading: localhost:5000/jiazha/openshift4-ose-sriov-network-device-plugin sha256:1912b7032cb3aa5b720caab583feb2a2f08262089ca409dce6c09a854e3cd307 16.37MiB
mounted: localhost:5000/jiazha/openshift4-ose-sriov-network-device-plugin sha256:ed6ee657d49e14dc574507ea575b857343d444d423231c7f827ae0d3105b7937 87.04MiB
sha256:b3e46c9b05e93da960ec6e1ba1923091b75f445e97b5fed10cff607cac370a99 localhost:5000/jiazha/openshift4-ose-sriov-network-device-plugin
sha256:aa710fd5fda4eefbe933231126f50c86f01886298e45615445b49b0797b31992 localhost:5000/jiazha/openshift4-ose-sriov-network-device-plugin
sha256:5f96dee0b0ea3efe5116e36043539e099abe3827a38fcb61b04c98b2eed27483 localhost:5000/jiazha/openshift4-ose-sriov-network-device-plugin
sha256:ffd968e2072085def6c5a78da1a0cdceeefc225253908947fec7763d7fc2f20f localhost:5000/jiazha/openshift4-ose-sriov-network-device-plugin
sha256:21c1db67da986eecf727ae9c47b5939bbbe0ac6e9a07b2ed23bd2a615280b5e9 localhost:5000/jiazha/openshift4-ose-sriov-network-device-plugin:55684a3b
info: Mirroring completed in 8.88s (84.59MB/s)
no digest mapping available for file://local/index/olmqe/sriov-fec:v4.9, skip writing to ImageContentSourcePolicy
wrote mirroring manifests to manifests-index/olmqe/sriov-fec-1644814454
deleted dir /tmp/1317676454

So, if CU will use Quay.io as the mirror registry in the future, I highly suggest that we update the media type of `sriov-fec` operator image to v2 from v1. 
For this bug, I will verify it since it can parse the old v1 version.

Updates:

> So, if CU will use Quay.io as the mirror registry in the future, I highly suggest that we update the media type of `sriov-fec` operator image to v2 from v1. 
For this bug, I will verify it since it can parse the old v1 version.

Please ignore it, confirm it with the Quay team, the Quay.io doesn't support the `application/vnd.oci.image.config.v1+json` media type, but Quay 3.6 supports it. That means if CU uses Quay 3.6 to create the registry, it will work well. Related doc: https://access.redhat.com/documentation/en-us/red_hat_quay/3/html-single/configure_red_hat_quay/index#other-oci-artifacts-with-quay

The last update on this was done on June 15th that this item was added added to advisory RHEA-2022:5069 by OpenShift Release Team Bot.



 @RossPeoples  When do we anticipate this bug being closed with an errata?

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069