Bug 1905095 - Images built on OCP 4.6 clusters create manifests that result in quay.io (and other registries) rejecting those manifests
Summary: Images built on OCP 4.6 clusters create manifests that result in quay.io (and...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 4.6
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.8.0
Assignee: Nalin Dahyabhai
QA Contact: XiuJuan Wang
: 1908908 1937433 (view as bug list)
Depends On:
Blocks: 1939218 1941452
TreeView+ depends on / blocked
Reported: 2020-12-07 13:40 UTC by Vadim Rutkovsky
Modified: 2021-07-27 22:35 UTC (History)
23 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2021-07-27 22:34:40 UTC
Target Upstream Version:

Attachments (Terms of Use)
oc adm mirror from quay to quay with v=5 (7.35 MB, text/plain)
2020-12-07 16:05 UTC, Vadim Rutkovsky
no flags Details
oc adm mirror from registry.svc to quay (8.23 MB, text/plain)
2020-12-07 16:57 UTC, Vadim Rutkovsky
no flags Details

System ID Private Priority Status Summary Last Updated
Github openshift builder pull 213 0 None closed Bug 1905095: bump github.com/containers/image to v5.10.1 2021-02-18 01:23:50 UTC
Github openshift builder pull 221 0 None open Bug 1905095: bump github.com/containers/image 2021-03-10 19:05:58 UTC
Github openshift origin pull 25830 0 None open Bug 1905095: Verify layer MIME types in images 2021-02-18 01:23:50 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 22:35:11 UTC

Description Vadim Rutkovsky 2020-12-07 13:40:48 UTC
Description of problem:
`oc adm release mirror` has started uploading some manifests as schema 1

Version-Release number of selected component (if applicable):
Latest from https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable/openshift-client-linux.tar.gz
Client Version: 4.6.6

How reproducible:

Steps to Reproduce:
1. Get pullspec of the image from payload:

$ RELEASE="4.6.0-0.okd-2020-12-07-083734"
$ oc adm release info --image-for aws-ebs-csi-driver-operator registry.svc.ci.openshift.org/origin/release:${RELEASE}

2. Check its schema version

$ skopeo inspect --authfile ~/src/github.com/vrutkovs/okd-installer/pull_secret.json docker://registry.svc.ci.openshift.org/origin/4.6-2020-12-07-083734@sha256:52f962cc969eaf5ab3c94e8c87eea75f12310dbde6ed3b0f7596c6d7200de08f --raw | jq '.schemaVersion'

(as expected)

3. Mirror the release:

$ oc adm -a /run/user/0/containers/auth.json release new --from-release registry.svc.ci.openshift.org/origin/release:${RELEASE} --mirror quay.io/openshift/okd-content --to-image quay.io/vrutkovs/okd-release:${RELEASE} --name=${RELEASE}

4. Check schema version of uploaded image:

$ oc adm release info --image-for aws-ebs-csi-driver-operator quay.io/vrutkovs/okd-release:${RELEASE}
$ skopeo inspect --authfile ~/src/github.com/vrutkovs/okd-installer/pull_secret.json docker://quay.io/vrutkovs/okd-release@sha256:ce862b5f752bea5fdd0d2c3b197c3b0362e02e3e433ad168bfc8ca380082e429 --raw | jq '.schemaVersion'

Actual results:
Some images are uploaded as schema 1 and throw signature verification errors during upload (as schema 1 digest doesn't match the expected)

Expected results:
All manifests are uploaded as schema 2

Additional info:

Comment 1 Vadim Rutkovsky 2020-12-07 16:05:46 UTC
Created attachment 1737353 [details]
oc adm mirror from quay to quay with v=5

Comment 2 Vadim Rutkovsky 2020-12-07 16:08:46 UTC
Comment on attachment 1737353 [details]
oc adm mirror from quay to quay with v=5

oc -v=5 adm -a ~/src/github.com/vrutkovs/okd-installer/pull_secret_quay.json release mirror \         
     --from=quay.io/openshift/okd:4.6.0-0.okd-2020-11-27-200126 \
     --to=quay.io/vrutkovs/okd-release \

Comment 3 Vadim Rutkovsky 2020-12-07 16:57:51 UTC
Created attachment 1737369 [details]
oc adm mirror from registry.svc to quay

>I1207 16:50:22.072264 1595705 manifest.go:498] Registry reported invalid manifest error, attempting to convert to v2schema1 as ref vrutkovs/okd-release:4.6.0- 0.okd-2020-12-07-083734-jenkins

Comment 4 Vadim Rutkovsky 2020-12-07 18:09:02 UTC
Seems Quay is rejecting some images and oc tries to push v2schema1:

I1207 17:57:09.238677 1620987 manifest.go:461] Put manifest vrutkovs/okd-release:4.6.0-0.okd-2020-12-07-083734-oauth-proxy
I1207 17:57:09.238757 1620987 round_trippers.go:423] curl -k -v -XPUT  -H "Content-Type: application/vnd.docker.distribution.manifest.v2+json" -H "Authorization: Bearer ..." 'https://quay.io/v2/vrutkovs/okd-release/manifests/4.6.0-0.okd-2020-12-07-083734-oauth-proxy'
I1207 17:57:09.336905 1620987 round_trippers.go:443] PUT https://quay.io/v2/vrutkovs/okd-release/manifests/4.6.0-0.okd-2020-12-07-083734-machine-config-operator 400 Bad Request in 98 milliseconds
I1207 17:57:09.337047 1620987 round_trippers.go:449] Response Headers:
I1207 17:57:09.337084 1620987 round_trippers.go:452]     Date: Mon, 07 Dec 2020 17:57:09 GMT
I1207 17:57:09.337130 1620987 round_trippers.go:452]     Content-Type: application/json
I1207 17:57:09.337160 1620987 round_trippers.go:452]     Content-Length: 979
I1207 17:57:09.337398 1620987 round_trippers.go:452]     Server: nginx/1.12.1
I1207 17:57:09.337900 1620987 manifest.go:498] Registry reported invalid manifest error, attempting to convert to v2schema1 as ref vrutkovs/okd-release:4.6.0-0.okd-2020-12-07-083734-machine-config-operator

Comment 5 Vadim Rutkovsky 2020-12-07 19:23:16 UTC
Apparently some images have broken manifest:
$ skopeo inspect docker://registry.ci.openshift.org/origin/4.6:tools
   "Env": [
$ oc -v=10 -a ~/src/github.com/vrutkovs/okd-installer/pull_secret_quay.json image mirror registry.ci.openshift.org/origin/4.6:tools quay.io/vrutkovs/okd-release
PUT https://quay.io/v2/vrutkovs/okd-release/manifests/tools 400 Bad Request in 207 milliseconds
This was built on 4.6.7

Some, like `registry.ci.openshift.org/origin/4.6:thanos` were built on 4.5 and are mirrored correctly

Comment 6 Vadim Rutkovsky 2020-12-07 19:44:47 UTC
{"errors":[{"code":"MANIFEST_INVALID","detail":{"message":"failed to parse manifest: manifest data does not match schema: u'application/vnd.docker.image.rootfs.diff.tar' is not one of ['application/vnd.docker.image.rootfs.diff.tar.gzip', 'application/vnd.docker.image.rootfs.foreign.diff.tar.gzip']\n\nFailed validating 'enum' in schema['properties']['layers']['items']['properties']['mediaType']:\n    {'description': 'The MIME type of the referenced object. This should generally be application/vnd.docker.image.rootfs.diff.tar.gzip. Layers of type application/vnd.docker.image.rootfs.foreign.diff.tar.gzip may be pulled from a remote location but they should never be pushed.',\n     'enum': ['applica…
from quay output, which seems to be


Comment 7 Gabe Montero 2020-12-07 20:06:22 UTC
Nalin has a PR up for https://github.com/containers/image/issues/733

Ultimately, we'll need to bump the buildah dep in openshift/builder to pick it up

Comment 8 Clayton Coleman 2020-12-07 20:08:07 UTC
Summary of issue:

1. OCP CI environments (4.6 OSD) import UBI and build a base image - that is done using buildah and the images already exist in the target registry (since base is super common), so the buildah bug is triggered
2. The base image is used as the parent for all CI images created on those clusters for all CI created releases (4.2 -> 4.8)
3. The CI tests use those images directly and appear to pull and mirror correctly within our CI clusters
4. The job that mirrors CI images to quay for OKD and developer access is rejected by quay for those images (rejects the application/vnd.docker.image.rootfs.diff.tar media type on the layer), which causes oc image mirror to fall back to a v2schema1 image, which causes the manifest to change
5. Payloads for OKD cannot be created for release that preserve SHAs because of this

It is too hard to spot these errors in the mirror command, so the mirror command should be printing the quay.io error message about the invalid manifest.

We need a fix to 4.6 and then rolled out to OSD in order to get out of this. Other components may fail to work with these images (although no evidence of that so far except a 3.11 registry reports a 500 when this image is pushed)

Comment 9 Adam Kaplan 2020-12-17 15:14:39 UTC
Progress on this issue is being made in https://github.com/containers/image/pull/1089

Comment 10 Adam Kaplan 2020-12-22 14:09:09 UTC
*** Bug 1908908 has been marked as a duplicate of this bug. ***

Comment 14 Nalin Dahyabhai 2021-01-14 16:42:47 UTC
Moving this back to assigned.  Once the we merge the upstream PR, we'll need a PR for the client, and then it'll be appropriate to move this back to POST, I think.

Comment 20 Adam Kaplan 2021-03-10 19:01:51 UTC
*** Bug 1937433 has been marked as a duplicate of this bug. ***

Comment 24 Adam Kaplan 2021-03-15 18:26:21 UTC
Current fix is awaiting CI verification. The following buildah dependencies are to be updated:

1. containers/image -> v5.10.5
2. containers/storage -> v1.24.8

Comment 25 XiuJuan Wang 2021-03-17 14:42:25 UTC
Could reproduce it with builder image in ocp 4.5(will check on 4.6 tomorrow).

Push a builder image,
$oc new-app ruby:latest~http://github.com/openshift/rails-ex.git

Check the image schemeVersion
$skopeo inspect docker://default-route-openshift-image-registry.apps.knarra4517.0317-3t7.qe.rhcloud.com/default/rails-ex:latest  --raw | jq '.schemaVersion'

Mirror image to quay.io
$oc image mirror default-route-openshift-image-registry.apps.knarra4517.0317-3t7.qe.rhcloud.com/default/rails-ex:latest quay.io/openshifttest/ocp:45 -a ~/.docker/quay-xiuwang

Check the quay.io image schemaVersion
$skopeo inspect docker://quay.io/openshifttest/ocp:45 --creds xiuwang:******* --raw | jq '.schemaVersion'

The bug fixs on ocp 4.8.0-0.nightly-2021-03-17-014745 cluster, validate it with same steps as above.

$skopeo inspect docker://default-route-openshift-image-registry.apps.xxia17.qe.devcluster.openshift.com/xiuwang/ruby-hello-world:latest  --raw | jq '.schemaVersion'

$oc image mirror default-route-openshift-image-registry.apps.xxia17.qe.devcluster.openshift.com/xiuwang/ruby-hello-world:latest quay.io/openshifttest/test:myimage -a ~/.docker/quay-xiuwang 
      default-route-openshift-image-registry.apps.xxia17.qe.devcluster.openshift.com/xiuwang/ruby-hello-world sha256:a858833a9239708c0c07c8fdf95218065c0605e14950051b009f9ad263f43511 1.765KiB
      default-route-openshift-image-registry.apps.xxia17.qe.devcluster.openshift.com/xiuwang/ruby-hello-world sha256:e10f2f601be71b985b78a7cb9002d952d8a30f8a5e526dd7265dbdc84b2da038 12.15KiB
      default-route-openshift-image-registry.apps.xxia17.qe.devcluster.openshift.com/xiuwang/ruby-hello-world sha256:608083cad0129a0f9240e5dcd4ceb087cc5ff025012277fc28bd77108e11a9bd 6.886MiB
      default-route-openshift-image-registry.apps.xxia17.qe.devcluster.openshift.com/xiuwang/ruby-hello-world sha256:c9caf8a93d59da90dcdfe05e92175baa22e7d6e443a95ffbbea3684e85bf46ba 10.55MiB
      default-route-openshift-image-registry.apps.xxia17.qe.devcluster.openshift.com/xiuwang/ruby-hello-world sha256:4f1355d64ea65ae6566038612e6e4d2d7384f5ac0323d7ca00f03cd84a9d6233 14MiB
      default-route-openshift-image-registry.apps.xxia17.qe.devcluster.openshift.com/xiuwang/ruby-hello-world sha256:b77f42d650dc7d0d6fa21f8661f03957cfe70fcf92e48245d2a7cad7d795eb56 72.89MiB
      default-route-openshift-image-registry.apps.xxia17.qe.devcluster.openshift.com/xiuwang/ruby-hello-world sha256:d15e5a5c8e28d36e53056a430b5aeb6a0d3fea187ebada478dd7cbb5524221bf 83.85MiB
      sha256:01c3e02a429e8096a9bb00653417691931562d260e3399d6b8520a020a613bae -> myimage
  stats: shared=0 unique=7 size=188.2MiB ratio=1.00

phase 0:
  quay.io openshifttest/test blobs=7 mounts=0 manifests=1 shared=0

info: Planning completed in 2.95s
uploading: quay.io/openshifttest/test sha256:c9caf8a93d59da90dcdfe05e92175baa22e7d6e443a95ffbbea3684e85bf46ba 10.55MiB
uploading: quay.io/openshifttest/test sha256:b77f42d650dc7d0d6fa21f8661f03957cfe70fcf92e48245d2a7cad7d795eb56 72.89MiB
uploading: quay.io/openshifttest/test sha256:4f1355d64ea65ae6566038612e6e4d2d7384f5ac0323d7ca00f03cd84a9d6233 14MiB
uploading: quay.io/openshifttest/test sha256:608083cad0129a0f9240e5dcd4ceb087cc5ff025012277fc28bd77108e11a9bd 6.886MiB
uploading: quay.io/openshifttest/test sha256:d15e5a5c8e28d36e53056a430b5aeb6a0d3fea187ebada478dd7cbb5524221bf 83.85MiB
sha256:01c3e02a429e8096a9bb00653417691931562d260e3399d6b8520a020a613bae quay.io/openshifttest/test:myimage
info: Mirroring completed in 5m48.76s (565.8kB/s)

$skopeo inspect docker://quay.io/openshifttest/test:myimage --creds xiuwang:******* --raw | jq '.schemaVersion' 

Comment 28 XiuJuan Wang 2021-03-18 06:59:59 UTC
Current fix is awaiting CI verification, the feature fixs are merged. 
Move to on_qa manually.

Comment 29 XiuJuan Wang 2021-03-18 07:05:15 UTC
Per comment #25, and do regression test for build features on ocp 4.8.0-0.nightly-2021-03-17-014745 cluster, no new issue found.
We could mark this bug as verified.

Comment 34 errata-xmlrpc 2021-07-27 22:34:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.