Description of problem: Follow the documentation https://docs.openshift.com/container-platform/4.5/operators/admin/olm-restricted-networks.html to sync all redhat operators. Try to install Cluster Logging, the Pod fail to start with: Back-off pulling image "registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766" Here the log of syncing only the cluster-logging images, followed the official documentation: $ podman run -ti --entrypoint bash ${LOCAL_REGISTRY}/olm/redhat-operators:${SERIAL} bash-4.2$ echo "select * from related_image \ where operatorbundle_name like 'clusterlogging.4.5%';" | sqlite3 -line /bundles.db image = registry.redhat.io/openshift4/ose-logging-fluentd@sha256:02b962d95d8d1c81e882e22ee215ab5639314e51ce4a5a3e997d244b05759f11 operatorbundle_name = clusterlogging.4.5.0-202010090328.p0 image = registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 operatorbundle_name = clusterlogging.4.5.0-202010090328.p0 image = registry.redhat.io/openshift4/ose-logging-curator5@sha256:c6456f5a66221cfcf269ae11af11b50a019b31daf382a4a0a41b560d9b631585 operatorbundle_name = clusterlogging.4.5.0-202010090328.p0 $ grep f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 redhat-operators-${SERIAL}/mapping.txt > redhat-operators-${SERIAL}/mapping.cluster-logging.txt $ grep 02b962d95d8d1c81e882e22ee215ab5639314e51ce4a5a3e997d244b05759f11 redhat-operators-${SERIAL}/mapping.txt >> redhat-operators-${SERIAL}/mapping.cluster-logging.txt $ grep c6456f5a66221cfcf269ae11af11b50a019b31daf382a4a0a41b560d9b631585 redhat-operators-${SERIAL}/mapping.txt >> redhat-operators-${SERIAL}/mapping.cluster-logging.txt $ cat redhat-operators-${SERIAL}/mapping.cluster-logging.txt registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766=host.compute.local:5000/openshift4/ose-cluster-logging-operator:79807487 registry.redhat.io/openshift4/ose-logging-fluentd@sha256:02b962d95d8d1c81e882e22ee215ab5639314e51ce4a5a3e997d244b05759f11=host.compute.local:5000/openshift4/ose-logging-fluentd:1d244428 registry.redhat.io/openshift4/ose-logging-curator5@sha256:c6456f5a66221cfcf269ae11af11b50a019b31daf382a4a0a41b560d9b631585=host.compute.local:5000/openshift4/ose-logging-curator5:5fca0599 $ oc image mirror -a ${LOCAL_SECRET_JSON} -f redhat-operators-${SERIAL}/mapping.cluster-logging.txt host.compute.local:5000/ openshift4/ose-cluster-logging-operator manifests: sha256:150885927c864c519664629cfb404f9dd279c7162562a370a0928b291012c6ce -> 79807487 openshift4/ose-logging-curator5 manifests: sha256:481e8e32fa8a47f942a59821442f593e9a406f550db2343728e1cb13755198bf -> 5fca0599 openshift4/ose-logging-fluentd manifests: sha256:cb81d753df19cfd7d9819831171eee68b186a5d15f1b178fcc7cc181678923af -> 1d244428 stats: shared=0 unique=0 size=0B phase 0: host.compute.local:5000 openshift4/ose-cluster-logging-operator blobs=0 mounts=0 manifests=1 shared=0 host.compute.local:5000 openshift4/ose-logging-fluentd blobs=0 mounts=0 manifests=1 shared=0 host.compute.local:5000 openshift4/ose-logging-curator5 blobs=0 mounts=0 manifests=1 shared=0 info: Planning completed in 1.48s sha256:150885927c864c519664629cfb404f9dd279c7162562a370a0928b291012c6ce host.compute.local:5000/openshift4/ose-cluster-logging-operator:79807487 sha256:481e8e32fa8a47f942a59821442f593e9a406f550db2343728e1cb13755198bf host.compute.local:5000/openshift4/ose-logging-curator5:5fca0599 sha256:cb81d753df19cfd7d9819831171eee68b186a5d15f1b178fcc7cc181678923af host.compute.local:5000/openshift4/ose-logging-fluentd:1d244428 info: Mirroring completed in 10ms (0B/s) It try to sync the operator image with sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 (mentioned in the csv), but the sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 forward to sha256:150885927c864c519664629cfb404f9dd279c7162562a370a0928b291012c6ce. Means the sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 is not available in our mirror registry. Skopoe shows it very well: $ skopeo --debug inspect --authfile ${LOCAL_SECRET_JSON} docker://registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 DEBU[0000] reference rewritten from 'registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766' to 'registry.redhat.io/openshift4/ose-cluster-loggin g-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766' DEBU[0000] Trying to pull "registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766" DEBU[0000] Returning credentials from /root/hetzner-ocp4/pullsecret.json DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration DEBU[0000] Using "default-docker" configuration DEBU[0000] No signature storage configuration found for registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io DEBU[0000] GET https://registry.redhat.io/v2/ DEBU[0000] Ping https://registry.redhat.io/v2/ status 401 DEBU[0000] GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?account=xxxxxxxx > DEBU[0000] GET https://registry.redhat.io/v2/openshift4/ose-cluster-logging-operator/manifests/sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 > DEBU[0001] GET https://registry.redhat.io/v2/openshift4/ose-cluster-logging-operator/manifests/sha256:150885927c864c519664629cfb404f9dd279c7162562a370a0928b291012c6ce ^^^^ ===== Looks like the "important" forward ===== DEBU[0001] Downloading /v2/openshift4/ose-cluster-logging-operator/blobs/sha256:e0226706a58476937aea5ef240e4a71268cf00b342925a7af4b124b80197c807 DEBU[0001] GET https://registry.redhat.io/v2/openshift4/ose-cluster-logging-operator/blobs/sha256:e0226706a58476937aea5ef240e4a71268cf00b342925a7af4b124b80197c807 DEBU[0001] Returning credentials from /root/hetzner-ocp4/pullsecret.json DEBU[0001] Using registries.d directory /etc/containers/registries.d for sigstore configuration DEBU[0001] Using "default-docker" configuration DEBU[0001] No signature storage configuration found for registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 DEBU[0001] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io DEBU[0001] GET https://registry.redhat.io/v2/ DEBU[0002] Ping https://registry.redhat.io/v2/ status 401 DEBU[0002] GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?account=52173376%7Cuhc-1ECxqMHPJD8Nu4qi56CMSM44hvz&scope=repository%3Aopenshift4%2Fose-cluster-logging-operator%3Apull&service=dock er-registry DEBU[0002] GET https://registry.redhat.io/v2/openshift4/ose-cluster-logging-operator/tags/list Version-Release number of selected component (if applicable): - OCP 4.5.2 - Cluster Logging: 4.5.0-202010090328.p0 How reproducible: Steps to Reproduce: 1. Install OCP4 Air-gapped - follow the documention 2. Sync redhat operators - follow the documentation 3. Install cluster logging operator Actual results: cluster-logging-operator pod fails to start with: Back-off pulling image "registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766" Expected results: cluster-logging-operator pod should start Additional info: My lab is available for tests, please let me know If I can support. The cluster-logging.v4.5.0.clusterserviceversion.yaml containers the failty sha: $ grep -A1 -B1 ose-cluster-logging-operator cluster-logging.v4.5.0.clusterserviceversion.yaml The Cluster Logging Operator for OKD provides a means for configuring and managing your aggregated logging stack. containerImage: registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 createdAt: 2018-08-01T08:00:00Z -- relatedImages: - name: ose-cluster-logging-operator image: registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 - name: ose-logging-curator5 -- - name: cluster-logging-operator image: registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 imagePullPolicy: IfNotPresent Source of the cluster-logging.v4.5.0.clusterserviceversion.yaml cache of "oc adm catalog build"
Runs os image mirror with -v=8, it looks like that is multiarch image problem: -v=8 hint: multi arch “forward” I1023 14:19:56.220387 583676 mirror.go:491] Found manifest sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 with type *manifestlist.DeserializedManifestList I1023 14:19:56.220408 583676 manifest.go:355] Including image for manifestlist.PlatformSpec{Architecture:"amd64", OS:"linux", OSVersion:"", OSFeatures:[]string(nil), Variant:"", Features:[]string(nil)} from registry.redhat.io/openshift4/ose-cluster-logging-operator I1023 14:19:56.220430 583676 manifest.go:352] Skipping image for manifestlist.PlatformSpec{Architecture:"ppc64le", OS:"linux", OSVersion:"", OSFeatures:[]string(nil), Variant:"", Features:[]string(nil)} from registry.redhat.io/openshift4/ose-cluster-logging-operator I1023 14:19:56.220444 583676 manifest.go:352] Skipping image for manifestlist.PlatformSpec{Architecture:"s390x", OS:"linux", OSVersion:"", OSFeatures:[]string(nil), Variant:"", Features:[]string(nil)} from registry.redhat.io/openshift4/ose-cluster-logging-operator I1023 14:19:56.220495 583676 manifest.go:379] Filtered manifest list to new digest sha256:077ee9ffb6146e26b751eb432aabe77339dd996a9cc0aaaf26ade67eb2eb4bba: { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json", "manifests": [ { "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 1371, "digest": "sha256:150885927c864c519664629cfb404f9dd279c7162562a370a0928b291012c6ce", "platform": { "architecture": "amd64", "os": "linux" } } ] } Work-a-round is to run oc image mirror or oc adm catalog mirror with --filter-by-os='.*' I don't know who is responsible for mirror stuff - might the OLM part.
--keep-manifest-list do not solve the problem, with 4.5.13 is --keep-manifest-list=false by default. $ oc version Client Version: 4.5.13 $ oc image mirror --help 2>&1 | grep keep-manifest --keep-manifest-list=false: If an image is part of a manifest list, always mirror the list even if only one image is found. The default is to mirror the specific image unless unless --filter-by-os is '.*'. $ cat hetzner-ocp4/redhat-operators-${SERIAL}/mapping-f90c4.txt registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766=host.compute.local:5000/openshift4/ose-cluster-logging-operator:79807487 $ skopeo inspect --authfile ${LOCAL_SECRET_JSON} docker://${LOCAL_REGISTRY}/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 | jq '.Name' FATA[0000] Error parsing image name "docker://host.compute.local:5000/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766": Error reading manifest sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 in host.compute.local:5000/openshift4/ose-cluster-logging-operator: manifest unknown: manifest unknown $ oc image mirror -a ${LOCAL_SECRET_JSON} -f redhat-operators-${SERIAL}/mapping-f90c4.txt --keep-manifest-list=true host.compute.local:5000/ openshift4/ose-cluster-logging-operator blobs: registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:2bd25ca124579d6fce8668ff5d4ed83866d7e7438cb561a51ddde8cc40272822 1.768KiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:e0226706a58476937aea5ef240e4a71268cf00b342925a7af4b124b80197c807 5.483KiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:de26e5fd8a5a0e38eaa7f357f0f0d3959280223022b6a40d0a62ec46cc863744 3.353MiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:e965397e401762a7c91d85ecf0b31d8ce5b0fd433048b653786aaec19f9dfd6f 9.155MiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:0c99d6c60bffe97c7d577e79d738047d711b1fc5b372c78d999d5bd08247db75 20.99MiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:1323a241cc068f2816dd88f00168be73339471d6dc6eb2e6c761b63b734501b6 72.85MiB manifests: sha256:077ee9ffb6146e26b751eb432aabe77339dd996a9cc0aaaf26ade67eb2eb4bba sha256:150885927c864c519664629cfb404f9dd279c7162562a370a0928b291012c6ce sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 -> 79807487 stats: shared=0 unique=6 size=106.4MiB ratio=1.00 phase 0: host.compute.local:5000 openshift4/ose-cluster-logging-operator blobs=6 mounts=0 manifests=3 shared=0 info: Planning completed in 1.42s uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:de26e5fd8a5a0e38eaa7f357f0f0d3959280223022b6a40d0a62ec46cc863744 3.353MiB uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:1323a241cc068f2816dd88f00168be73339471d6dc6eb2e6c761b63b734501b6 72.85MiB uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:e965397e401762a7c91d85ecf0b31d8ce5b0fd433048b653786aaec19f9dfd6f 9.155MiB uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:0c99d6c60bffe97c7d577e79d738047d711b1fc5b372c78d999d5bd08247db75 20.99MiB sha256:150885927c864c519664629cfb404f9dd279c7162562a370a0928b291012c6ce host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:077ee9ffb6146e26b751eb432aabe77339dd996a9cc0aaaf26ade67eb2eb4bba host.compute.local:5000/openshift4/ose-cluster-logging-operator:79807487 error: unable to push manifest to host.compute.local:5000/openshift4/ose-cluster-logging-operator: manifest blob unknown: blob unknown to registry info: Mirroring completed in 7.63s (14.6MB/s) error: one or more errors occurred while uploading images $ skopeo inspect --authfile ${LOCAL_SECRET_JSON} docker://${LOCAL_REGISTRY}/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 | jq '.Name' FATA[0000] Error parsing image name "docker://host.compute.local:5000/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766": Error reading manifest sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 in host.compute.local:5000/openshift4/ose-cluster-logging-operator: manifest unknown: manifest unknown Only add --filter-by-os=".*" solve the problem: $ oc image mirror -a ${LOCAL_SECRET_JSON} -f redhat-operators-${SERIAL}/mapping-f90c4.txt --keep-manifest-list=true --filter-by-os=".*" host.compute.local:5000/ openshift4/ose-cluster-logging-operator blobs: registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:2bd25ca124579d6fce8668ff5d4ed83866d7e7438cb561a51ddde8cc40272822 1.768KiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:d1b57d73d9c8c74ed27f33a527e3008d7ddea4328b719df90bc74e1fadd069e9 1.784KiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:ea26005206bce5d3cfac2da18b76ebd27865b39c5aa9b4a2c0e78a4842bba3b6 1.793KiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:db58e12a850c0fc9eed05dc47557608080f6a9a11a007e9044befda75bddfc30 5.482KiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:e0226706a58476937aea5ef240e4a71268cf00b342925a7af4b124b80197c807 5.483KiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:e3f1476ea3864481132101529a327282d03ce7c48f414916b533d3a83c21871e 5.494KiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:de26e5fd8a5a0e38eaa7f357f0f0d3959280223022b6a40d0a62ec46cc863744 3.353MiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:523e1ad755e7dae4a72c92ee466af48b8a3f740a3fd1682b776519038a6675ff 3.611MiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:81d7c66c2a31d6196074b5f39e32a10a6edaa997d526fbab40ea2f9b11be8d8b 4.189MiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:e965397e401762a7c91d85ecf0b31d8ce5b0fd433048b653786aaec19f9dfd6f 9.155MiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:5f88f8a633957a79091938fb5fd47516253f941f63c2421098572c1badd1ee34 9.704MiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:54682c4eb4192cd24ce12b88009758c1bc4e73dca9a03e8480f0f1d90166022f 9.818MiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:ad20b01de7b73810b971e23bd0cb410a80f397fd9598002f39fc2fe7b777207f 19.14MiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:0c99d6c60bffe97c7d577e79d738047d711b1fc5b372c78d999d5bd08247db75 20.99MiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:a9a8f817082ad17ccbc857524cd5f6be7cf4a90bfce035eca05611250a8e216c 21.52MiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:1323a241cc068f2816dd88f00168be73339471d6dc6eb2e6c761b63b734501b6 72.85MiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:184b56835891110cbed643cdfd628d5b770069be584d83bfbe6fb0666c87645b 73.27MiB registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:77a0bc0be79a60e619bff47345e7937c222a3705cff3cf9f90e17fbd5927f0cb 76.86MiB manifests: sha256:150885927c864c519664629cfb404f9dd279c7162562a370a0928b291012c6ce sha256:96056c0aeec9e950a03e638318b26d6e52caace77023e32d18ddbd7775ebdd10 sha256:d40817849b0f18e2839ed2d8641950ce80538d0958630167aff81c31f2cde89c sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 -> 79807487 stats: shared=0 unique=18 size=324.5MiB ratio=1.00 phase 0: host.compute.local:5000 openshift4/ose-cluster-logging-operator blobs=18 mounts=0 manifests=5 shared=0 info: Planning completed in 4.38s uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:a9a8f817082ad17ccbc857524cd5f6be7cf4a90bfce035eca05611250a8e216c 21.52MiB uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:523e1ad755e7dae4a72c92ee466af48b8a3f740a3fd1682b776519038a6675ff 3.611MiB uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:77a0bc0be79a60e619bff47345e7937c222a3705cff3cf9f90e17fbd5927f0cb 76.86MiB uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:184b56835891110cbed643cdfd628d5b770069be584d83bfbe6fb0666c87645b 73.27MiB uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:81d7c66c2a31d6196074b5f39e32a10a6edaa997d526fbab40ea2f9b11be8d8b 4.189MiB uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:54682c4eb4192cd24ce12b88009758c1bc4e73dca9a03e8480f0f1d90166022f 9.818MiB uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:5f88f8a633957a79091938fb5fd47516253f941f63c2421098572c1badd1ee34 9.704MiB uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:ad20b01de7b73810b971e23bd0cb410a80f397fd9598002f39fc2fe7b777207f 19.14MiB sha256:96056c0aeec9e950a03e638318b26d6e52caace77023e32d18ddbd7775ebdd10 host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:150885927c864c519664629cfb404f9dd279c7162562a370a0928b291012c6ce host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:d40817849b0f18e2839ed2d8641950ce80538d0958630167aff81c31f2cde89c host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 host.compute.local:5000/openshift4/ose-cluster-logging-operator:79807487 info: Mirroring completed in 9.35s (24.44MB/s) $ skopeo inspect --authfile ${LOCAL_SECRET_JSON} docker://${LOCAL_REGISTRY}/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 | jq '.Name' "host.compute.local:5000/openshift4/ose-cluster-logging-operator"
*** Bug 1891549 has been marked as a duplicate of this bug. ***
I am updating the Severity and Priority of this ticket due to the number of cases that are being reported.
*** Bug 1898853 has been marked as a duplicate of this bug. ***
Hi, We do have a customer hitting the same issue, However none of the workaround suggest works for customer.Please let us know if there is any other workaround. Thanks, Dixit
Created attachment 1735497 [details] Bug1890951-tests-with-ocp4.6.6.details.log The problem still exists with 4.6.6. Here my simplified summary, all the details attached. During the sync with oc adm catalog mirror \ ${LOCAL_REGISTRY}/redhat/redhat-operator-index:v${OCP_RELEASE%\.*} \ ${LOCAL_REGISTRY} \ it sync the manifest list, BUT it removes the not needed architectures. This breaks the digest of the manifest list by design! But the digest of the full manifest list is used for the operator deployment: image: >- registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:540b0d087c5e5529bab555030310478630249a9339a4a4c3fac6d0d7037d5eac Here the details of the debug sync log: I1202 06:40:03.920447 2096345 mirror.go:513] Found manifest sha256:540b0d087c5e5529bab555030310478630249a9339a4a4c3fac6d0d7037d5eac with type *manifestlist.DeserializedManifestList I1202 06:40:03.920465 2096345 manifest.go:373] Including image sha256:5518705b8d98dcb6dfbfb2ba5b375ec052944cdea8667f9f26cbe5a713780118 for manifestlist.PlatformSpec{Architecture:"amd64", OS:"linux", OSVersion:"", OSFeatures:[]string(nil), Variant:"", Features:[]string(nil)} from registry.redhat.io/openshift4/ose-cluster-logging-operator I1202 06:40:03.920481 2096345 manifest.go:370] Skipping image sha256:5b3fa37a4fba08eddc92a13ed45e469f6d3d089f4580fff35646e507f15620bd for manifestlist.PlatformSpec{Architecture:"ppc64le", OS:"linux", OSVersion:"", OSFeatures:[]string(nil), Variant:"", Features:[]string(nil)} from registry.redhat.io/openshift4/ose-cluster-logging-operator I1202 06:40:03.920491 2096345 manifest.go:370] Skipping image sha256:7ffe5e80d9ffcfc2d6fd47add4bc110ccec8ae4a8232f6bdd25bea30e9202656 for manifestlist.PlatformSpec{Architecture:"s390x", OS:"linux", OSVersion:"", OSFeatures:[]string(nil), Variant:"", Features:[]string(nil)} from registry.redhat.io/openshift4/ose-cluster-logging-operator I1202 06:40:03.920525 2096345 manifest.go:397] Filtered manifest list to new digest sha256:236052852d9a5bca319e7734c041d7cd18679f21dbca4dbce87aee615b415074: { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json", "manifests": [ { "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 1372, "digest": "sha256:5518705b8d98dcb6dfbfb2ba5b375ec052944cdea8667f9f26cbe5a713780118", "platform": { "architecture": "amd64", "os": "linux" } } ] Ergo sha256:540b0d087c5e5529bab555030310478630249a9339a4a4c3fac6d0d7037d5eac is not available anymore. That looks like a design problem, from my understanding you can not push a full manifest list unless all other architectures digests are available at the target registry. As ecordell mention [1] In case all the multiarch manifests stuff is new to you, I wrote a blogpost about it [2] because it was also new to me :-) The work-a-round to add `--filter-by-os='.*'` still works for me. [1] https://github.com/openshift/oc/pull/642#issuecomment-736758745 [2] https://www.opensourcerers.org/2020/11/16/container-images-multi-architecture-manifests-ids-digests-whats-behind/
Thank Robert, Had a call with customer today and we were able to resolve the issue using `--filter-by-os='.*'` , basically new catalog image built using "oc adm catalog build" was not getting picked during catalogsoure deployment since an old image existed on the node and it was not pulling the new image. After "oc adm prune images" and removing all older catalogsource we were able to deploy a new catalogsource and then cluster logging operator installed successfully. Regards, Dixit
*** Bug 1903230 has been marked as a duplicate of this bug. ***
Also, there's an open PR to address confusion over --filter-by-os=.* vs --keep-manifest-list=true, once that PR merges I'll update here since I'm not sure what the decision will be regarding these flags, PR is under review. (openshift oc pull 642)
Merged OCP docs PR for the --filter-by-os='.*' workaround: https://github.com/openshift/openshift-docs/pull/27855 Updated doc has been published (step 2, callout 4): https://docs.openshift.com/container-platform/4.5/operators/admin/olm-restricted-networks.html#olm-restricted-networks-operatorhub_olm-restricted-networks Same for 4.6.
Something is not right with the --filter-by-os='.*' filter in OpenShift 4.6: https://docs.openshift.com/container-platform/4.6/operators/admin/olm-restricted-networks.html#olm-mirror-catalog_olm-restricted-networks ~~~ oc adm catalog mirror \ <index_image> \ <mirror_registry>:<port> \ [-a ${REG_CREDS}] \ [--insecure] \ --filter-by-os='.*' \ [--manifests-only] ~~~ The doc clearly states: ~~~ This flag is currently required due to a known issue with multiple architecture support. If unset or set to any value other than .*, filtering out different architectures changes the digest of the manifest list, also known as a "multi-arch image", which causes deployments of those images and Operators on disconnected clusters to fail. For more information, see BZ#1890951. ~~~ When I run this, I get: ~~~ [root@openshift-jumpserver-0 ~]# oc adm catalog mirror registry.redhat.io/redhat/redhat-operator-index:v4.6 192.168.123.1:5000 -a ${REG_CREDS} --filter-by-os='.*' error: the image is a manifest list and contains multiple images - use --filter-by-os to select from: OS DIGEST linux/amd64 sha256:7d1e513ffad86c02189de25479aebef62ca4cef7d588cb3910ae1791c1537d78 linux/ppc64le sha256:14ca1b701ba1d95b8de22100738325469442a8aaf6d655ddfe4ca511b7fb07af linux/s390x sha256:ccac6405ca2658c03c6f262fd23ef7c97584d2e1f676eaba31943110b99fbff7 [root@openshift-jumpserver-0 ~]# oc adm catalog mirror registry.redhat.io/redhat/redhat-operator-index:v4.6 192.168.123.1:5000 -a ${REG_CREDS} --filter-by-os=.* error: the image is a manifest list and contains multiple images - use --filter-by-os to select from: OS DIGEST linux/amd64 sha256:7d1e513ffad86c02189de25479aebef62ca4cef7d588cb3910ae1791c1537d78 linux/ppc64le sha256:14ca1b701ba1d95b8de22100738325469442a8aaf6d655ddfe4ca511b7fb07af linux/s390x sha256:ccac6405ca2658c03c6f262fd23ef7c97584d2e1f676eaba31943110b99fbff7 ~~~
That's with: [root@openshift-jumpserver-0 ~]# oc version Client Version: 4.6.1 Server Version: 4.6.1
I updated the client to 4.6.8 and still see the same: [root@openshift-jumpserver-0 ~]# oc adm catalog mirror registry.redhat.io/redhat/redhat-operator-index:v4.6 192.168.123.1:5000 -a ${REG_CREDS} --filter-by-os=.* error: the image is a manifest list and contains multiple images - use --filter-by-os to select from: OS DIGEST linux/amd64 sha256:7d1e513ffad86c02189de25479aebef62ca4cef7d588cb3910ae1791c1537d78 linux/ppc64le sha256:14ca1b701ba1d95b8de22100738325469442a8aaf6d655ddfe4ca511b7fb07af linux/s390x sha256:ccac6405ca2658c03c6f262fd23ef7c97584d2e1f676eaba31943110b99fbff7 [root@openshift-jumpserver-0 ~]# oc adm catalog mirror registry.redhat.io/redhat/redhat-operator-index:v4.6 192.168.123.1:5000 -a ${REG_CREDS} --filter-by-os=".*" error: the image is a manifest list and contains multiple images - use --filter-by-os to select from: OS DIGEST linux/amd64 sha256:7d1e513ffad86c02189de25479aebef62ca4cef7d588cb3910ae1791c1537d78 linux/ppc64le sha256:14ca1b701ba1d95b8de22100738325469442a8aaf6d655ddfe4ca511b7fb07af linux/s390x sha256:ccac6405ca2658c03c6f262fd23ef7c97584d2e1f676eaba31943110b99fbff7 [root@openshift-jumpserver-0 ~]# oc version Client Version: 4.6.8 Server Version: 4.6.1 Kubernetes Version: v1.19.0+d59ce34
I am using OCP 4.6.9 and the latest version of oc cli. I am getting the same error message like Andreas with --filter-by-os='.*' parameter. [core@bastiontest olm]$ oc adm catalog mirror registry.redhat.io/redhat/redhat-operator-index:v4.6 scorenexus.isbank:9443/olmr -a olm_pull-secret --filter-by-os='.*' --manifests-only error: the image is a manifest list and contains multiple images - use --filter-by-os to select from: OS DIGEST linux/amd64 sha256:1b13f5d739219cf4aaf2578e1fe455bb6e1ac58c545cacfe2efd38c721e6ff85 linux/ppc64le sha256:b5954897d5487b956097a094fbf444dd4d84dfa056206286adc2cd49b69ebf3d linux/s390x sha256:8804c0d52b9ff5fab9357b89415e08f0462d1db15306f579b0e09dc938864e27 [core@bastiontest olm]$ oc version Client Version: openshift-clients-4.6.0-202006250705.p0-168-g02c110006 Server Version: 4.6.9 Kubernetes Version: v1.19.0+7070803
I am getting this on OC 4.6.15 client. Please advise. [root@bastion ~]# oc adm catalog mirror registry.redhat.io/redhat/redhat-operator-index:v4.6 bastion.airgap1.cdastu.com:5000 -a ${REG_CREDS} --filter-by-os=".*" --insecure error: the image is a manifest list and contains multiple images - use --filter-by-os to select from: OS DIGEST linux/amd64 sha256:804d289905c20889e50aec6728a85020d8f30ca46f88d84e34683cef2918eb18 linux/ppc64le sha256:78eee018c0c3db14201ccbef4084836564080c9258a97427ca24b018ba9823e0 linux/s390x sha256:805ceb30b17ccd942178626480df1cb60f99706792b0f79ecd88c7aedffae194 [root@bastion ~]# oc version Client Version: 4.6.15 Server Version: 4.6.15 Kubernetes Version: v1.19.0+1833054
Hi Stu, You should just no longer set filter-by-os in the latest version of oc. By default, it will select an arch option if you don't set it. The flag is still there for backwards compatibility reasons, but for the latest images selecting .* will never work because the index image itself has multiple architectures and one has to be picked to pull out the catalog contents from it.
OK, that worked. I assume that there will be a Doc change on this?