Bug 1890677

Summary:	'oc image info' claims 'does not exist' for application/vnd.oci.image.manifest.v1+json manifest
Product:	OpenShift Container Platform	Reporter:	W. Trevor King <wking>
Component:	oc	Assignee:	Sally <somalley>
Status:	CLOSED ERRATA	QA Contact:	zhou ying <yinzhou>
Severity:	low	Docs Contact:
Priority:	low
Version:	4.5	CC:	aos-bugs, jokerman, mfojtik, sjenning
Target Milestone:	---	Keywords:	UpcomingSprint
Target Release:	4.7.0	Flags:	mfojtik: needinfo?
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	LifecycleReset
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1990006 (view as bug list)		Environment:
Last Closed:	2021-02-24 15:27:41 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1990006

Description W. Trevor King 2020-10-22 17:26:55 UTC

Description of problem:

For a public image with a application/vnd.oci.image.manifest.v1+json manifest, 'oc image info $PULLSPEC' and other oc tooling claim 'does not exist' instead of successfully accessing the manifest.

Version-Release number of selected component (if applicable):

$ oc version --client
Client Version: 4.5.11

How reproducible:

Every time.

Steps to Reproduce:

$ oc --v=8 image info docker.io/sjenning/olm:dev-5

Actual results:

...
I1022 10:23:05.575427   13423 round_trippers.go:420] GET https://registry-1.docker.io/v2/sjenning/olm/manifests/dev-5
I1022 10:23:05.575476   13423 round_trippers.go:427] Request Headers:
I1022 10:23:05.575512   13423 round_trippers.go:431]     Accept: application/vnd.docker.distribution.manifest.v1+prettyjws
I1022 10:23:05.575547   13423 round_trippers.go:431]     Accept: application/json
I1022 10:23:05.575594   13423 round_trippers.go:431]     Accept: application/vnd.docker.distribution.manifest.v2+json
I1022 10:23:05.575628   13423 round_trippers.go:431]     Accept: application/vnd.docker.distribution.manifest.list.v2+json
I1022 10:23:05.575668   13423 round_trippers.go:431]     Accept: application/vnd.oci.image.index.v1+json
I1022 10:23:05.575722   13423 round_trippers.go:431]     Authorization: Bearer <masked>
I1022 10:23:05.868414   13423 round_trippers.go:446] Response Status: 404 Not Found in 292 milliseconds
I1022 10:23:05.868465   13423 round_trippers.go:449] Response Headers:
I1022 10:23:05.868485   13423 round_trippers.go:452]     Content-Type: application/json
I1022 10:23:05.868501   13423 round_trippers.go:452]     Docker-Distribution-Api-Version: registry/2.0
I1022 10:23:05.868517   13423 round_trippers.go:452]     Date: Thu, 22 Oct 2020 17:23:05 GMT
I1022 10:23:05.868532   13423 round_trippers.go:452]     Content-Length: 122
I1022 10:23:05.868546   13423 round_trippers.go:452]     Strict-Transport-Security: max-age=31536000
I1022 10:23:05.868952   13423 workqueue.go:143] about to send work queue error: image does not exist
F1022 10:23:05.869081   13423 helpers.go:115] error: image does not exist

Expected results:

Display manifest metadata.

Additional info:

Pulling with skopeo:

$ skopeo --debug inspect docker://docker.io/sjenning/olm:dev-5
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]  No signature storage configuration found for docker.io/sjenning/olm:dev-5 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io 
DEBU[0000] GET https://registry-1.docker.io/v2/         
DEBU[0000] Ping https://registry-1.docker.io/v2/ err <nil> 
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 
DEBU[0000] GET https://registry-1.docker.io/v2/sjenning/olm/manifests/dev-5 
DEBU[0001] Downloading /v2/sjenning/olm/blobs/sha256:1b22c48e0eb5f6aa7ec169079aa9d208a7e4276899eb135a472dbc35a9e6b2b1 
DEBU[0001] GET https://registry-1.docker.io/v2/sjenning/olm/blobs/sha256:1b22c48e0eb5f6aa7ec169079aa9d208a7e4276899eb135a472dbc35a9e6b2b1 
DEBU[0001] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0001]  Using "default-docker" configuration        
DEBU[0001]  No signature storage configuration found for docker.io/sjenning/olm:dev-5 
DEBU[0001] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io 
DEBU[0001] GET https://registry-1.docker.io/v2/         
DEBU[0002] Ping https://registry-1.docker.io/v2/ err <nil> 
DEBU[0002] Ping https://registry-1.docker.io/v2/ status 401 
DEBU[0002] GET https://registry-1.docker.io/v2/sjenning/olm/tags/list 
{
    "Name": "docker.io/sjenning/olm",
...

And directly with curl:

$ TOKEN=$(curl -s 'https://auth.docker.io/token?scope=repository%3Asjenning%2Folm%3Apull&service=registry.docker.io' | jq -r .access_token)
$ curl -sH "Authorization: Bearer ${TOKEN}" https://registry-1.docker.io/v2/sjenning/olm/manifests/dev-5
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"OCI manifest found, but accept header does not support OCI manifests"}]}
$ curl -sH "Authorization: Bearer ${TOKEN}" -H 'Accept:application/vnd.oci.image.manifest.v1+json' https://registry-1.docker.io/v2/sjenning/olm/manifests/dev-5 | jq .
{
  "schemaVersion": 2,
...

I dunno why oc does not appear to include application/vnd.oci.image.manifest.v1+json when making requests of the manifests/{tag} endpoint.

Comment 1 Seth Jennings 2020-10-22 17:40:41 UTC

I deployed my own registry using docker's registry:2 image, pushed the image build with podman to the registry and then `oc image info`ed it.  It failed.  The log from the registry side is:

level=error msg="response completed with error" err.code="manifest unknown" err.message="OCI manifest found, but accept header does not support OCI manifests"

Comment 3 Sally 2020-11-12 17:40:56 UTC

Actively working on this, adding UpcomingSprint.

Comment 4 Michal Fojtik 2020-11-21 19:12:05 UTC

This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.

Comment 5 Sally 2020-12-05 00:25:09 UTC

There are many schema 1 images on quay that do work fine with `oc image info`, for example here are a few:

quay.io/openshift/origin-cluster-version-operator:4.6.0
quay.io/openshift/origin-console:4.6.0
quay.io/openshift/origin-console-tests:4.6.0
quay.io/openshift/origin-docker-builder:4.6.0

With the 'Media Type:  application/vnd.docker.distribution.manifest.v1+prettyjws'  

Then, I can do this (also running w/ the local podman registry image) :

$ oc image mirror quay.io/openshift/origin-docker-builder:4.6.0=localhost:5000/docker-builder:4.6.0

and then this works fine

$ oc image info localhost:5000/docker-builder:4.6.0

Does this work for you? 

`oc` does handle schema version 1 images, it will upconvert them to v2 so when you mirror a v1 you'll end up w/ a new digest and you'll see a warning:
"warning: Digests are not preserved with schema version 1 images. Support for schema version 1 images will be removed in a future release" but I don't have any trouble accessing them with `oc image info`.

Comment 6 W. Trevor King 2020-12-05 00:41:20 UTC

With my random, master-ish arm64 build of oc available on my current machine:

$ oc version --client
Client Version: v4.2.0-alpha.0-873-ge575833
$ oc --v=8 image info quay.io/openshift/origin-cluster-version-operator:4.6.0
...
I1204 16:37:58.121770   12439 round_trippers.go:420] HEAD https://quay.io/v2/openshift/origin-cluster-version-operator/manifests/4.6.0
I1204 16:37:58.121786   12439 round_trippers.go:427] Request Headers:
I1204 16:37:58.121801   12439 round_trippers.go:431]     Accept: application/json
I1204 16:37:58.121815   12439 round_trippers.go:431]     Accept: application/vnd.docker.distribution.manifest.v2+json
I1204 16:37:58.121829   12439 round_trippers.go:431]     Accept: application/vnd.docker.distribution.manifest.list.v2+json
I1204 16:37:58.121843   12439 round_trippers.go:431]     Accept: application/vnd.oci.image.index.v1+json
I1204 16:37:58.121856   12439 round_trippers.go:431]     Accept: application/vnd.docker.distribution.manifest.v1+prettyjws
I1204 16:37:58.121873   12439 round_trippers.go:431]     Authorization: Bearer <masked>
I1204 16:37:58.252114   12439 round_trippers.go:446] Response Status: 200 OK in 130 milliseconds
I1204 16:37:58.252206   12439 round_trippers.go:449] Response Headers:
I1204 16:37:58.252253   12439 round_trippers.go:452]     X-Frame-Options: DENY
I1204 16:37:58.252299   12439 round_trippers.go:452]     Strict-Transport-Security: max-age=63072000; preload
I1204 16:37:58.252341   12439 round_trippers.go:452]     Server: nginx/1.12.1
I1204 16:37:58.252381   12439 round_trippers.go:452]     Date: Sat, 05 Dec 2020 00:37:59 GMT
I1204 16:37:58.252424   12439 round_trippers.go:452]     Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
...

That's quay.io saying "sure, I can serve that to you in application/vnd.docker.distribution.manifest.v1+prettyjws, which is one of the types you can Accept".  But if we were talking to a registry about a manifest that could only be served as application/vnd.oci.image.manifest.v1+json , oc would be out of luck, right?

Comment 7 Michal Fojtik 2020-12-05 01:13:37 UTC

The LifecycleStale keyword was removed because the bug got commented on recently.
The bug assignee was notified.

Comment 8 Michal Fojtik 2021-01-04 01:58:21 UTC

This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.

Comment 9 Sally 2021-01-12 14:15:26 UTC

I'm getting back to this bug now, was set aside due to other bugs with higher priority. In the least, oc should provide a more accurate error message. I'll also look into why oc can't accept the OCI manifest. For example, when I try to mirror that image I see this (a bit better) error message:

$ oc image mirror docker.io/sjenning/olm:dev-5=localhost:5000/olm:dev

error: unable to retrieve source image docker.io/sjenning/olm by tag dev-5: manifest unknown: OCI manifest found, but accept header does not support OCI manifests
error: an error occurred during planning

Comment 10 Michal Fojtik 2021-01-12 14:38:52 UTC

The LifecycleStale keyword was removed because the bug got commented on recently.
The bug assignee was notified.

Comment 12 zhou ying 2021-01-19 08:17:58 UTC

Can't reproduce the issue with latest oc client:
[root@preserver-workloadrhel-1 ~]# oc version --client
Client Version: 4.7.0-0.nightly-2021-01-19-014259


[root@preserver-workloadrhel-1 ~]# oc  image info docker.io/sjenning/olm:dev-5
Name:        docker.io/sjenning/olm:dev-5
Digest:      sha256:93fe89549a4ece5e92df915069d0554e8759debdb3f09846deca45dcd4dd9beb
Media Type:  application/vnd.oci.image.manifest.v1+json
Created:     95d ago
Image Size:  184.7MB in 10 layers
Layers:      78.15MB sha256:b80ee16c866200b7aca5ae763b95a878e756c7bbbd7cc3b19a033bf1372efc61
             1.805kB sha256:6eeb9b4a640ff7b7b8bbac12b72740c753337a5d020fd5ebc1d9244a787ca7db
             5.652MB sha256:a49b5ee785371f6e8f603fae831288ea7df31a5ef81fb3b1ab142e7833d70ff1
             470.4kB sha256:1917f31fd5f9b57925ba2da55bf55b49f15279f9cde8f1729e0993aa723daadb
             11.48MB sha256:172c84f160d84f280736a94b1036869d5b4475da91492e9c0e88bfa2126afc0f
             87.69kB sha256:1dba7b14c3de2af464a2eba8bfd54a339ee444bf525e3d0c759c04bbe7c81dca
             24.34MB sha256:44e894a6f99ad3b5bef85a3c8b1f161cb0bf616f287094552ab8b2e651c4588a
             23.39MB sha256:6cb078b7541d4f6a754ba9d4bf8c5d46387f42f7f658ba9834585b872a35d0f0
             24.95MB sha256:c4d107990cb4704f5d58cc7d2c6fc5ce10fdad807c6ea4b6ca6eb0b2f5a2b533
             16.18MB sha256:386d6e423f988b58eaed001f34eaed50c5dde4d108ef9f94ee6c2cffef9525a7
OS:          linux
Arch:        amd64
Command:     /bin/bash
User:        0
Environment: foo=bar
             GODEBUG=x509ignoreCN=0
             OPENSHIFT_BUILD_NAME=base-8
             OPENSHIFT_BUILD_NAMESPACE=ci-op-7sb1cdz6
             OPENSHIFT_CI=true
             PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
             container=oci
Labels:      architecture=x86_64
             build-date=2020-09-05T01:13:15.933978
             com.redhat.build-host=cpt-1003.osbs.prod.upshift.rdu2.redhat.com
             com.redhat.component=openshift-enterprise-base-container
             com.redhat.license_terms=https://www.redhat.com/agreements
             description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.
             distribution-scope=public
             io.buildah.version=1.16.1
             io.k8s.description=This is a component of OpenShift Container Platform and manages the lifecycle of operators.
             io.k8s.display-name=OpenShift Operator Lifecycle Manager
             io.openshift.build.commit.author=
             io.openshift.build.commit.date=
             io.openshift.build.commit.id=ec6a8a871fe24f5fb6cfbc9f35a9c3c1d5633458
             io.openshift.build.commit.message=
             io.openshift.build.commit.ref=master
             io.openshift.build.name=
             io.openshift.build.namespace=
             io.openshift.build.source-context-dir=
             io.openshift.build.source-location=https://github.com/openshift/images
             io.openshift.expose-services=
             io.openshift.release.operator=true
             io.openshift.tags=base rhel8
             maintainer=Odin Team <aos-odin>
             name=openshift/ose-base
             release=202009050041.5133
             summary=Provides the latest release of Red Hat Universal Base Image 8.
             url=https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-base/images/v4.0-202009050041.5133
             vcs-ref=ec6a8a871fe24f5fb6cfbc9f35a9c3c1d5633458
             vcs-type=git
             vcs-url=https://github.com/openshift/images
             vendor=Red Hat, Inc.
             version=v4.0

Comment 15 errata-xmlrpc 2021-02-24 15:27:41 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633