Description of problem: Failed to mirror image to intergrated registry with "manifest invalid" error sometimes. Version-Release number of selected component (if applicable): $oc version Client Version: 4.4.0-0.nightly-2020-03-03-033819 Server Version: 4.4.0-0.nightly-2020-03-03-011650 Kubernetes Version: v1.17.1 How reproducible: 20% Steps to Reproduce: 1.Mirror image cat $imagelist.yaml registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cluster-logging-operator@sha256:47623e1873c86d9c1bdd41cfb06ae7f7f175a1f74364c441c4d6dd765369059d=default-route-openshift-image-registry.apps.qeci-3214.0303-otz.qe.rhcloud.com/openshift/ose-cluster-logging-operator:v4.4.0 $oc image mirror --filename=imagelist.yaml --insecure=true -a docker-config.yaml --skip-missing=true --skip-verification=true 2. 3. Actual results: $oc image mirror --filename=imagelist.yaml --insecure=true -a docker-config.yaml --skip-missing=true --skip-verification=true default-route-openshift-image-registry.apps.qeci-3214.0303-otz.qe.rhcloud.com/ openshift/ose-cluster-logging-operator blobs: registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cluster-logging-operator sha256:455ea8ab06218495bbbcb14b750a0d644897b24f8c5dcf9e8698e27882583412 1.575KiB registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cluster-logging-operator sha256:0146a450aa03211b02a411edc2123bdf156bc17808d2f554111c7b068a36c63e 5.351KiB registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cluster-logging-operator sha256:935ce2f796a92f0b6fa7762d7bab7d252ddb59d03cd3afba42d3785b571854f1 3.333MiB registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cluster-logging-operator sha256:b6555363e7f53679e058c70fcda710f30ba4f6e457645c1025c154b539ecd038 7.847MiB registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cluster-logging-operator sha256:b6f8660c86d2591f34712e27797fdd5e51b7ebd410610d16160a06da81791d56 22.04MiB registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cluster-logging-operator sha256:bb13d92caffa705f32b8a7f9f661e07ddede310c6ccfa78fb53a49539740e29b 72.71MiB manifests: sha256:5856b8f00dcb7c4b23c25028535f4085adbfd73ff105f69b993f2f7ea1080d1e sha256:917f7f7a0ba92a6ed1293ea54a7486fee71fa53e1cd25b23586c457ad402709d sha256:47623e1873c86d9c1bdd41cfb06ae7f7f175a1f74364c441c4d6dd765369059d -> v4.4.0 stats: shared=0 unique=6 size=105.9MiB ratio=1.00 phase 0: default-route-openshift-image-registry.apps.qeci-3214.0303-otz.qe.rhcloud.com openshift/ose-cluster-logging-operator blobs=6 mounts=0 manifests=3 shared=0 info: Planning completed in 5.4s error: unable to push manifest to default-route-openshift-image-registry.apps.qeci-3214.0303-otz.qe.rhcloud.com/openshift/ose-cluster-logging-operator: received unexpected HTTP status: 500 Internal Server Error error: unable to push manifest to default-route-openshift-image-registry.apps.qeci-3214.0303-otz.qe.rhcloud.com/openshift/ose-cluster-logging-operator: manifest invalid: manifest invalid error: unable to push manifest to default-route-openshift-image-registry.apps.qeci-3214.0303-otz.qe.rhcloud.com/openshift/ose-cluster-logging-operator:v4.4.0: manifest invalid: manifest invalid info: Mirroring completed in 3.15s (0B/s) error: one or more errors occurred while uploading images Expected results: Should mirror registry succeed. Additional info: Could succeed with docker push this image. The image-registry pod log during this cmd failed, http://pastebin.test.redhat.com/841342
That is 100% reproduced now. set high priority.
The images need to be mirrored to intenral cluster for 3rd Operators testing on public cloud. That is blocking QE automation testing. That will block the custom to deploy 3rd level operators to disconnect cluster. (The workaround is easy. Use 4.3.4 and and prior oc)
Looks like the fix from https://github.com/openshift/oc/pull/298 might be the cause of this problem, moving to OLM folks for further investigation.
Hi, Maciej > Looks like the fix from https://github.com/openshift/oc/pull/298 might be the cause of this problem, moving to OLM folks for further investigation. Yes, you're right! In this commit, we force to mirror the manifest as well. But, the cluster internal registry doesn't support the v2-2(app registry). So, it will fail. Now, we add a flag `--force-manifest-list` to allow that option. PR: https://github.com/openshift/oc/pull/339
Image mirror worked as expected. Marking as verified oc version Client Version: 4.4.0-0.nightly-2020-03-06-141620 Steps used to reproduce. 1) Started a local registry as oriented on https://bugzilla.redhat.com/show_bug.cgi?id=1773821#c7 2) cat imagelist.yml registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cluster-logging-operator@sha256:47623e1873c86d9c1bdd41cfb06ae7f7f175a1f74364c441c4d6dd765369059d=localhost:5000/test:v4.4.0 3) oc image mirror --filename=imagelist.yml --insecure=true --skip-missing=true --skip-verification=true localhost:5000/ test blobs: registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cluster-logging-operator sha256:455ea8ab06218495bbbcb14b750a0d644897b24f8c5dcf9e8698e27882583412 1.575KiB registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cluster-logging-operator sha256:0146a450aa03211b02a411edc2123bdf156bc17808d2f554111c7b068a36c63e 5.351KiB registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cluster-logging-operator sha256:935ce2f796a92f0b6fa7762d7bab7d252ddb59d03cd3afba42d3785b571854f1 3.333MiB registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cluster-logging-operator sha256:b6555363e7f53679e058c70fcda710f30ba4f6e457645c1025c154b539ecd038 7.847MiB registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cluster-logging-operator sha256:b6f8660c86d2591f34712e27797fdd5e51b7ebd410610d16160a06da81791d56 22.04MiB registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cluster-logging-operator sha256:bb13d92caffa705f32b8a7f9f661e07ddede310c6ccfa78fb53a49539740e29b 72.71MiB manifests: sha256:917f7f7a0ba92a6ed1293ea54a7486fee71fa53e1cd25b23586c457ad402709d -> v4.4.0 stats: shared=0 unique=6 size=105.9MiB ratio=1.00 phase 0: localhost:5000 test blobs=6 mounts=0 manifests=1 shared=0 info: Planning completed in 2.04s uploading: localhost:5000/test sha256:935ce2f796a92f0b6fa7762d7bab7d252ddb59d03cd3afba42d3785b571854f1 3.333MiB uploading: localhost:5000/test sha256:bb13d92caffa705f32b8a7f9f661e07ddede310c6ccfa78fb53a49539740e29b 72.71MiB uploading: localhost:5000/test sha256:b6555363e7f53679e058c70fcda710f30ba4f6e457645c1025c154b539ecd038 7.847MiB uploading: localhost:5000/test sha256:b6f8660c86d2591f34712e27797fdd5e51b7ebd410610d16160a06da81791d56 22.04MiB sha256:917f7f7a0ba92a6ed1293ea54a7486fee71fa53e1cd25b23586c457ad402709d localhost:5000/test:v4.4.0 info: Mirroring completed in 1m28.87s (1.25MB/s)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581