Description of problem: For a public image with a application/vnd.oci.image.manifest.v1+json manifest, 'oc image info $PULLSPEC' and other oc tooling claim 'does not exist' instead of successfully accessing the manifest. Version-Release number of selected component (if applicable): $ oc version --client Client Version: 4.5.11 How reproducible: Every time. Steps to Reproduce: $ oc --v=8 image info docker.io/sjenning/olm:dev-5 Actual results: ... I1022 10:23:05.575427 13423 round_trippers.go:420] GET https://registry-1.docker.io/v2/sjenning/olm/manifests/dev-5 I1022 10:23:05.575476 13423 round_trippers.go:427] Request Headers: I1022 10:23:05.575512 13423 round_trippers.go:431] Accept: application/vnd.docker.distribution.manifest.v1+prettyjws I1022 10:23:05.575547 13423 round_trippers.go:431] Accept: application/json I1022 10:23:05.575594 13423 round_trippers.go:431] Accept: application/vnd.docker.distribution.manifest.v2+json I1022 10:23:05.575628 13423 round_trippers.go:431] Accept: application/vnd.docker.distribution.manifest.list.v2+json I1022 10:23:05.575668 13423 round_trippers.go:431] Accept: application/vnd.oci.image.index.v1+json I1022 10:23:05.575722 13423 round_trippers.go:431] Authorization: Bearer <masked> I1022 10:23:05.868414 13423 round_trippers.go:446] Response Status: 404 Not Found in 292 milliseconds I1022 10:23:05.868465 13423 round_trippers.go:449] Response Headers: I1022 10:23:05.868485 13423 round_trippers.go:452] Content-Type: application/json I1022 10:23:05.868501 13423 round_trippers.go:452] Docker-Distribution-Api-Version: registry/2.0 I1022 10:23:05.868517 13423 round_trippers.go:452] Date: Thu, 22 Oct 2020 17:23:05 GMT I1022 10:23:05.868532 13423 round_trippers.go:452] Content-Length: 122 I1022 10:23:05.868546 13423 round_trippers.go:452] Strict-Transport-Security: max-age=31536000 I1022 10:23:05.868952 13423 workqueue.go:143] about to send work queue error: image does not exist F1022 10:23:05.869081 13423 helpers.go:115] error: image does not exist Expected results: Display manifest metadata. Additional info: Pulling with skopeo: $ skopeo --debug inspect docker://docker.io/sjenning/olm:dev-5 DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration DEBU[0000] Using "default-docker" configuration DEBU[0000] No signature storage configuration found for docker.io/sjenning/olm:dev-5 DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io DEBU[0000] GET https://registry-1.docker.io/v2/ DEBU[0000] Ping https://registry-1.docker.io/v2/ err <nil> DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 DEBU[0000] GET https://registry-1.docker.io/v2/sjenning/olm/manifests/dev-5 DEBU[0001] Downloading /v2/sjenning/olm/blobs/sha256:1b22c48e0eb5f6aa7ec169079aa9d208a7e4276899eb135a472dbc35a9e6b2b1 DEBU[0001] GET https://registry-1.docker.io/v2/sjenning/olm/blobs/sha256:1b22c48e0eb5f6aa7ec169079aa9d208a7e4276899eb135a472dbc35a9e6b2b1 DEBU[0001] Using registries.d directory /etc/containers/registries.d for sigstore configuration DEBU[0001] Using "default-docker" configuration DEBU[0001] No signature storage configuration found for docker.io/sjenning/olm:dev-5 DEBU[0001] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io DEBU[0001] GET https://registry-1.docker.io/v2/ DEBU[0002] Ping https://registry-1.docker.io/v2/ err <nil> DEBU[0002] Ping https://registry-1.docker.io/v2/ status 401 DEBU[0002] GET https://registry-1.docker.io/v2/sjenning/olm/tags/list { "Name": "docker.io/sjenning/olm", ... And directly with curl: $ TOKEN=$(curl -s 'https://auth.docker.io/token?scope=repository%3Asjenning%2Folm%3Apull&service=registry.docker.io' | jq -r .access_token) $ curl -sH "Authorization: Bearer ${TOKEN}" https://registry-1.docker.io/v2/sjenning/olm/manifests/dev-5 {"errors":[{"code":"MANIFEST_UNKNOWN","message":"OCI manifest found, but accept header does not support OCI manifests"}]} $ curl -sH "Authorization: Bearer ${TOKEN}" -H 'Accept:application/vnd.oci.image.manifest.v1+json' https://registry-1.docker.io/v2/sjenning/olm/manifests/dev-5 | jq . { "schemaVersion": 2, ... I dunno why oc does not appear to include application/vnd.oci.image.manifest.v1+json when making requests of the manifests/{tag} endpoint.
I deployed my own registry using docker's registry:2 image, pushed the image build with podman to the registry and then `oc image info`ed it. It failed. The log from the registry side is: level=error msg="response completed with error" err.code="manifest unknown" err.message="OCI manifest found, but accept header does not support OCI manifests"
Actively working on this, adding UpcomingSprint.
This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.
There are many schema 1 images on quay that do work fine with `oc image info`, for example here are a few: quay.io/openshift/origin-cluster-version-operator:4.6.0 quay.io/openshift/origin-console:4.6.0 quay.io/openshift/origin-console-tests:4.6.0 quay.io/openshift/origin-docker-builder:4.6.0 With the 'Media Type: application/vnd.docker.distribution.manifest.v1+prettyjws' Then, I can do this (also running w/ the local podman registry image) : $ oc image mirror quay.io/openshift/origin-docker-builder:4.6.0=localhost:5000/docker-builder:4.6.0 and then this works fine $ oc image info localhost:5000/docker-builder:4.6.0 Does this work for you? `oc` does handle schema version 1 images, it will upconvert them to v2 so when you mirror a v1 you'll end up w/ a new digest and you'll see a warning: "warning: Digests are not preserved with schema version 1 images. Support for schema version 1 images will be removed in a future release" but I don't have any trouble accessing them with `oc image info`.
With my random, master-ish arm64 build of oc available on my current machine: $ oc version --client Client Version: v4.2.0-alpha.0-873-ge575833 $ oc --v=8 image info quay.io/openshift/origin-cluster-version-operator:4.6.0 ... I1204 16:37:58.121770 12439 round_trippers.go:420] HEAD https://quay.io/v2/openshift/origin-cluster-version-operator/manifests/4.6.0 I1204 16:37:58.121786 12439 round_trippers.go:427] Request Headers: I1204 16:37:58.121801 12439 round_trippers.go:431] Accept: application/json I1204 16:37:58.121815 12439 round_trippers.go:431] Accept: application/vnd.docker.distribution.manifest.v2+json I1204 16:37:58.121829 12439 round_trippers.go:431] Accept: application/vnd.docker.distribution.manifest.list.v2+json I1204 16:37:58.121843 12439 round_trippers.go:431] Accept: application/vnd.oci.image.index.v1+json I1204 16:37:58.121856 12439 round_trippers.go:431] Accept: application/vnd.docker.distribution.manifest.v1+prettyjws I1204 16:37:58.121873 12439 round_trippers.go:431] Authorization: Bearer <masked> I1204 16:37:58.252114 12439 round_trippers.go:446] Response Status: 200 OK in 130 milliseconds I1204 16:37:58.252206 12439 round_trippers.go:449] Response Headers: I1204 16:37:58.252253 12439 round_trippers.go:452] X-Frame-Options: DENY I1204 16:37:58.252299 12439 round_trippers.go:452] Strict-Transport-Security: max-age=63072000; preload I1204 16:37:58.252341 12439 round_trippers.go:452] Server: nginx/1.12.1 I1204 16:37:58.252381 12439 round_trippers.go:452] Date: Sat, 05 Dec 2020 00:37:59 GMT I1204 16:37:58.252424 12439 round_trippers.go:452] Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws ... That's quay.io saying "sure, I can serve that to you in application/vnd.docker.distribution.manifest.v1+prettyjws, which is one of the types you can Accept". But if we were talking to a registry about a manifest that could only be served as application/vnd.oci.image.manifest.v1+json , oc would be out of luck, right?
The LifecycleStale keyword was removed because the bug got commented on recently. The bug assignee was notified.
I'm getting back to this bug now, was set aside due to other bugs with higher priority. In the least, oc should provide a more accurate error message. I'll also look into why oc can't accept the OCI manifest. For example, when I try to mirror that image I see this (a bit better) error message: $ oc image mirror docker.io/sjenning/olm:dev-5=localhost:5000/olm:dev error: unable to retrieve source image docker.io/sjenning/olm by tag dev-5: manifest unknown: OCI manifest found, but accept header does not support OCI manifests error: an error occurred during planning
Can't reproduce the issue with latest oc client: [root@preserver-workloadrhel-1 ~]# oc version --client Client Version: 4.7.0-0.nightly-2021-01-19-014259 [root@preserver-workloadrhel-1 ~]# oc image info docker.io/sjenning/olm:dev-5 Name: docker.io/sjenning/olm:dev-5 Digest: sha256:93fe89549a4ece5e92df915069d0554e8759debdb3f09846deca45dcd4dd9beb Media Type: application/vnd.oci.image.manifest.v1+json Created: 95d ago Image Size: 184.7MB in 10 layers Layers: 78.15MB sha256:b80ee16c866200b7aca5ae763b95a878e756c7bbbd7cc3b19a033bf1372efc61 1.805kB sha256:6eeb9b4a640ff7b7b8bbac12b72740c753337a5d020fd5ebc1d9244a787ca7db 5.652MB sha256:a49b5ee785371f6e8f603fae831288ea7df31a5ef81fb3b1ab142e7833d70ff1 470.4kB sha256:1917f31fd5f9b57925ba2da55bf55b49f15279f9cde8f1729e0993aa723daadb 11.48MB sha256:172c84f160d84f280736a94b1036869d5b4475da91492e9c0e88bfa2126afc0f 87.69kB sha256:1dba7b14c3de2af464a2eba8bfd54a339ee444bf525e3d0c759c04bbe7c81dca 24.34MB sha256:44e894a6f99ad3b5bef85a3c8b1f161cb0bf616f287094552ab8b2e651c4588a 23.39MB sha256:6cb078b7541d4f6a754ba9d4bf8c5d46387f42f7f658ba9834585b872a35d0f0 24.95MB sha256:c4d107990cb4704f5d58cc7d2c6fc5ce10fdad807c6ea4b6ca6eb0b2f5a2b533 16.18MB sha256:386d6e423f988b58eaed001f34eaed50c5dde4d108ef9f94ee6c2cffef9525a7 OS: linux Arch: amd64 Command: /bin/bash User: 0 Environment: foo=bar GODEBUG=x509ignoreCN=0 OPENSHIFT_BUILD_NAME=base-8 OPENSHIFT_BUILD_NAMESPACE=ci-op-7sb1cdz6 OPENSHIFT_CI=true PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=oci Labels: architecture=x86_64 build-date=2020-09-05T01:13:15.933978 com.redhat.build-host=cpt-1003.osbs.prod.upshift.rdu2.redhat.com com.redhat.component=openshift-enterprise-base-container com.redhat.license_terms=https://www.redhat.com/agreements description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly. distribution-scope=public io.buildah.version=1.16.1 io.k8s.description=This is a component of OpenShift Container Platform and manages the lifecycle of operators. io.k8s.display-name=OpenShift Operator Lifecycle Manager io.openshift.build.commit.author= io.openshift.build.commit.date= io.openshift.build.commit.id=ec6a8a871fe24f5fb6cfbc9f35a9c3c1d5633458 io.openshift.build.commit.message= io.openshift.build.commit.ref=master io.openshift.build.name= io.openshift.build.namespace= io.openshift.build.source-context-dir= io.openshift.build.source-location=https://github.com/openshift/images io.openshift.expose-services= io.openshift.release.operator=true io.openshift.tags=base rhel8 maintainer=Odin Team <aos-odin> name=openshift/ose-base release=202009050041.5133 summary=Provides the latest release of Red Hat Universal Base Image 8. url=https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-base/images/v4.0-202009050041.5133 vcs-ref=ec6a8a871fe24f5fb6cfbc9f35a9c3c1d5633458 vcs-type=git vcs-url=https://github.com/openshift/images vendor=Red Hat, Inc. version=v4.0
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633