I'd like to request a backport of https://bugzilla.redhat.com/show_bug.cgi?id=1890677 into OCP 4.6, as the issue can still be observed there. I found this when following the instructions from here https://access.redhat.com/solutions/5565971 but with OCP 4.6, hence: ~~~ podman pull registry.redhat.io/redhat/redhat-operator-index:v4.7 --authfile pull-secret.json podman run --name opm -d registry.redhat.io/redhat/redhat-operator-index:v4.7 podman commit opm registry.example.com:5000/opm-testing/redhat-operator-index:v4.7 ~~~ Podman will commit the image as ~~~ [root@openshift-jumpserver-0 ~]# podman inspect registry.example.com:5000/opm-testing/redhat-operator-index:v4.7 | grep -i manifest "ManifestType": "application/vnd.oci.image.manifest.v1+json", ~~~ Now, push the image: ~~~ podman push registry.example.com:5000/opm-testing/redhat-operator-index:v4.7 ~~~ And inspect it - with 4.7, this works, with 4.6.41, this does not work: ~~~ [root@openshift-jumpserver-0 ~]# oc version Client Version: 4.7.11 Server Version: 4.7.11 Kubernetes Version: v1.20.0+75370d3 [root@openshift-jumpserver-0 ~]# oc image info registry.example.com:5000/opm-testing/redhat-operator-index:v4.7 | head Name: registry.example.com:5000/opm-testing/redhat-operator-index:v4.7 Digest: sha256:84ada7d60d8fff9e7b6dff4ad4c4eee2e37fff1a7d97602eed46027ade5785f8 Media Type: application/vnd.oci.image.manifest.v1+json Created: 3m ago Image Size: 285MB in 7 layers Layers: 88.21MB sha256:e87c84ead535462834169e0ce6a487bc5dc1733ee99956013d33c7bef4dca9be 1.874kB sha256:dd1301fbf37f0ba2bf2e015866f2edb53e4e53b78587c9d0e0ae2ac3683a1dc4 13.9MB sha256:ad1a9e2134f17302b107c9fc0b439fb5d98c10c1cf8eb835cc628b27059798a5 11.53MB sha256:023c5e67a396a2486dc0b5b542d7e10bc714644dc4d6e5af1f67e267070b66db 151.8MB sha256:d87cef2a6a34aa3573de956d6b32212020101da8b42958a2fb021136e7d6ff47 [root@openshift-jumpserver-0 ~]# ./oc version Client Version: 4.6.41 Server Version: 4.7.11 Kubernetes Version: v1.20.0+75370d3 [root@openshift-jumpserver-0 ~]# ./oc image info registry.example.com:5000/opm-testing/redhat-operator-index:v4.7 error: image "registry.example.com:5000/opm-testing/redhat-operator-index:v4.7" does not exist ~~~ The problem is due to the requested headers: 4.7: ~~~ [root@openshift-jumpserver-0 ~]# oc --loglevel=10 image info registry.example.com:5000/opm-testing/redhat-operator-index:v4.7 2>&1 | grep curl I0804 14:47:57.904103 276138 round_trippers.go:425] curl -k -v -XGET 'https://registry.example.com:5000/v2/' I0804 14:47:57.948013 276138 round_trippers.go:425] curl -k -v -XHEAD -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" -H "Accept: application/vnd.oci.image.index.v1+json" -H "Accept: application/vnd.oci.image.manifest.v1+json" -H "Accept: application/vnd.docker.distribution.manifest.v1+prettyjws" -H "Accept: application/json" 'https://registry.example.com:5000/v2/opm-testing/redhat-operator-index/manifests/v4.7' I0804 14:47:57.950023 276138 round_trippers.go:425] curl -k -v -XGET -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Accept: application/vnd.oci.image.manifest.v1+json" 'https://registry.example.com:5000/v2/opm-testing/redhat-operator-index/manifests/sha256:84ada7d60d8fff9e7b6dff4ad4c4eee2e37fff1a7d97602eed46027ade5785f8' I0804 14:47:57.951953 276138 round_trippers.go:425] curl -k -v -XGET -H "Accept-Encoding: identity" 'https://registry.example.com:5000/v2/opm-testing/redhat-operator-index/blobs/sha256:fbc3199c1957487ac6aa35f5f403771f84f4ea3de462b08a84297f0c4c4e2f9f' ~~~ ~~~ [root@openshift-jumpserver-0 ~]# curl -k -v -XHEAD -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" -H "Accept: application/vnd.oci.image.index.v1+json" -H "Accept: application/vnd.oci.image.manifest.v1+json" -H "Accept: application/vnd.docker.distribution.manifest.v1+prettyjws" -H "Accept: application/json" 'https://registry.example.com:5000/v2/opm-testing/redhat-operator-index/manifests/v4.7' 2>&1 | grep 'HTTP/2' * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 > HEAD /v2/opm-testing/redhat-operator-index/manifests/v4.7 HTTP/2 < HTTP/2 200 ~~~ 4.6: ~~~ [root@openshift-jumpserver-0 ~]# ./oc --loglevel=10 image info registry.example.com:5000/opm-testing/redhat-operator-index:v4.7 2>&1 | grep curl I0804 14:47:59.407812 276147 round_trippers.go:423] curl -k -v -XGET 'https://registry.example.com:5000/v2/' I0804 14:47:59.454891 276147 round_trippers.go:423] curl -k -v -XHEAD -H "User-Agent: oc/4.6.0 (linux/amd64) kubernetes/2b525e8" -H "Accept: application/vnd.oci.image.index.v1+json" -H "Accept: application/vnd.docker.distribution.manifest.v1+prettyjws" -H "Accept: application/json" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" 'https://registry.example.com:5000/v2/opm-testing/redhat-operator-index/manifests/v4.7' I0804 14:47:59.460603 276147 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" -H "Accept: application/vnd.oci.image.index.v1+json" -H "Accept: application/vnd.docker.distribution.manifest.v1+prettyjws" -H "Accept: application/json" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "User-Agent: oc/4.6.0 (linux/amd64) kubernetes/2b525e8" 'https://registry.example.com:5000/v2/opm-testing/redhat-operator-index/manifests/v4.7' ~~~ ~~~ [root@openshift-jumpserver-0 ~]# curl -k -v -XHEAD -H "User-Agent: oc/4.6.0 (linux/amd64) kubernetes/2b525e8" -H "Accept: application/vnd.oci.image.index.v1+json" -H "Accept: application/vnd.docker.distribution.manifest.v1+prettyjws" -H "Accept: application/json" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" 'https://registry.example.com:5000/v2/opm-testing/redhat-operator-index/manifests/v4.7' 2>&1 | grep 'HTTP/2' * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 > HEAD /v2/opm-testing/redhat-operator-index/manifests/v4.7 HTTP/2 < HTTP/2 404 ~~~
Ross the problem with this one is that we don't support v1 schema, and that's perfectly fine for write operation, but for backwareds compatibility we should still read it. So the following operation should work: $ oc image mirror k8s.gcr.io/stress:v1 quay.io/repo/stress:v1 error: unable to retrieve source image k8s.gcr.io/stress manifest sha256:f00aa1ddc963a3164aef741aab0fc05074ea96de6cd7e0d10077cf98dd72d594: manifest unknown: Manifest with digest 'sha256:f00aa1ddc963a3164aef741aab0fc05074ea96de6cd7e0d10077cf98dd72d594' has media type 'application/vnd.docker.distribution.manifest.v1+prettyjws', but client accepts 'application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json'. error: an error occurred during planning but as seen above it fails b/c oc does not know about schema v1.