Bug 1890951 - Mirror of multiarch images together with cluster logging case problems. It doesn't sync the "overall" sha it syncs only the sub arch sha.
Summary: Mirror of multiarch images together with cluster logging case problems. It do...
Keywords:
Status: CLOSED DUPLICATE of bug 1908565
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc
Version: 4.5
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: 4.7.0
Assignee: Sally
QA Contact: zhou ying
URL:
Whiteboard:
: 1891549 1898853 (view as bug list)
Depends On:
Blocks: 1723620
TreeView+ depends on / blocked
 
Reported: 2020-10-23 11:41 UTC by Robert Bohne
Modified: 2021-02-23 15:21 UTC (History)
33 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-01-12 14:37:17 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Bug1890951-tests-with-ocp4.6.6.details.log (11.67 KB, text/plain)
2020-12-02 07:07 UTC, Robert Bohne
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift oc pull 642 0 None closed Bug 1890951: Improve output of 'oc image mirror' with multi-arch images 2021-02-21 13:45:51 UTC
Red Hat Bugzilla 1879877 0 high CLOSED oc adm catalog mirror --filter-by-os="linux/amd64" doesn't mirror all necessary images in "Using Operator Lifecycle Mana... 2023-12-15 19:23:47 UTC
Red Hat Bugzilla 1882689 0 high CLOSED c adm catalog mirror --filter-by-os="linux/amd64" doesn't mirror all necessary images in "Using Operator Lifecycle Manag... 2023-12-15 19:31:44 UTC
Red Hat Issue Tracker RFE-1324 0 Medium Backlog Provides a mechanism to avoid mirroring unnecessary architecture's images 2020-12-14 15:57:05 UTC

Internal Links: 1879877

Description Robert Bohne 2020-10-23 11:41:43 UTC
Description of problem:

Follow the documentation https://docs.openshift.com/container-platform/4.5/operators/admin/olm-restricted-networks.html to sync all redhat operators. Try to install Cluster Logging, the Pod fail to start with: 

Back-off pulling image "registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766"

Here the log of syncing only the cluster-logging images, followed the official documentation:

$ podman run -ti --entrypoint bash ${LOCAL_REGISTRY}/olm/redhat-operators:${SERIAL}
bash-4.2$ echo "select * from related_image \
    where operatorbundle_name like 'clusterlogging.4.5%';"     | sqlite3 -line /bundles.db
              image = registry.redhat.io/openshift4/ose-logging-fluentd@sha256:02b962d95d8d1c81e882e22ee215ab5639314e51ce4a5a3e997d244b05759f11
operatorbundle_name = clusterlogging.4.5.0-202010090328.p0
              image = registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766
operatorbundle_name = clusterlogging.4.5.0-202010090328.p0
              image = registry.redhat.io/openshift4/ose-logging-curator5@sha256:c6456f5a66221cfcf269ae11af11b50a019b31daf382a4a0a41b560d9b631585
operatorbundle_name = clusterlogging.4.5.0-202010090328.p0
$ grep f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 redhat-operators-${SERIAL}/mapping.txt  > redhat-operators-${SERIAL}/mapping.cluster-logging.txt
$ grep 02b962d95d8d1c81e882e22ee215ab5639314e51ce4a5a3e997d244b05759f11 redhat-operators-${SERIAL}/mapping.txt  >> redhat-operators-${SERIAL}/mapping.cluster-logging.txt
$ grep c6456f5a66221cfcf269ae11af11b50a019b31daf382a4a0a41b560d9b631585 redhat-operators-${SERIAL}/mapping.txt  >> redhat-operators-${SERIAL}/mapping.cluster-logging.txt
$ cat redhat-operators-${SERIAL}/mapping.cluster-logging.txt
registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766=host.compute.local:5000/openshift4/ose-cluster-logging-operator:79807487
registry.redhat.io/openshift4/ose-logging-fluentd@sha256:02b962d95d8d1c81e882e22ee215ab5639314e51ce4a5a3e997d244b05759f11=host.compute.local:5000/openshift4/ose-logging-fluentd:1d244428
registry.redhat.io/openshift4/ose-logging-curator5@sha256:c6456f5a66221cfcf269ae11af11b50a019b31daf382a4a0a41b560d9b631585=host.compute.local:5000/openshift4/ose-logging-curator5:5fca0599
$ oc image mirror -a ${LOCAL_SECRET_JSON} -f redhat-operators-${SERIAL}/mapping.cluster-logging.txt
host.compute.local:5000/
  openshift4/ose-cluster-logging-operator
    manifests:
      sha256:150885927c864c519664629cfb404f9dd279c7162562a370a0928b291012c6ce -> 79807487
  openshift4/ose-logging-curator5
    manifests:
      sha256:481e8e32fa8a47f942a59821442f593e9a406f550db2343728e1cb13755198bf -> 5fca0599
  openshift4/ose-logging-fluentd
    manifests:
      sha256:cb81d753df19cfd7d9819831171eee68b186a5d15f1b178fcc7cc181678923af -> 1d244428
  stats: shared=0 unique=0 size=0B
phase 0:
  host.compute.local:5000 openshift4/ose-cluster-logging-operator blobs=0 mounts=0 manifests=1 shared=0
  host.compute.local:5000 openshift4/ose-logging-fluentd          blobs=0 mounts=0 manifests=1 shared=0
  host.compute.local:5000 openshift4/ose-logging-curator5         blobs=0 mounts=0 manifests=1 shared=0
info: Planning completed in 1.48s
sha256:150885927c864c519664629cfb404f9dd279c7162562a370a0928b291012c6ce host.compute.local:5000/openshift4/ose-cluster-logging-operator:79807487
sha256:481e8e32fa8a47f942a59821442f593e9a406f550db2343728e1cb13755198bf host.compute.local:5000/openshift4/ose-logging-curator5:5fca0599
sha256:cb81d753df19cfd7d9819831171eee68b186a5d15f1b178fcc7cc181678923af host.compute.local:5000/openshift4/ose-logging-fluentd:1d244428
info: Mirroring completed in 10ms (0B/s)



It try to sync the operator image with sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 (mentioned in the csv), but the sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 forward to sha256:150885927c864c519664629cfb404f9dd279c7162562a370a0928b291012c6ce. Means the sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 is not available in our mirror registry.

Skopoe shows it very well:

$ skopeo --debug inspect --authfile ${LOCAL_SECRET_JSON} docker://registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766
DEBU[0000] reference rewritten from 'registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766' to 'registry.redhat.io/openshift4/ose-cluster-loggin
g-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766'
DEBU[0000] Trying to pull "registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766"
DEBU[0000] Returning credentials from /root/hetzner-ocp4/pullsecret.json
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration
DEBU[0000]  Using "default-docker" configuration
DEBU[0000]  No signature storage configuration found for registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io
DEBU[0000] GET https://registry.redhat.io/v2/
DEBU[0000] Ping https://registry.redhat.io/v2/ status 401
DEBU[0000] GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?account=xxxxxxxx


> DEBU[0000] GET https://registry.redhat.io/v2/openshift4/ose-cluster-logging-operator/manifests/sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766
> DEBU[0001] GET https://registry.redhat.io/v2/openshift4/ose-cluster-logging-operator/manifests/sha256:150885927c864c519664629cfb404f9dd279c7162562a370a0928b291012c6ce
^^^^ ===== Looks like the "important" forward =====


DEBU[0001] Downloading /v2/openshift4/ose-cluster-logging-operator/blobs/sha256:e0226706a58476937aea5ef240e4a71268cf00b342925a7af4b124b80197c807
DEBU[0001] GET https://registry.redhat.io/v2/openshift4/ose-cluster-logging-operator/blobs/sha256:e0226706a58476937aea5ef240e4a71268cf00b342925a7af4b124b80197c807
DEBU[0001] Returning credentials from /root/hetzner-ocp4/pullsecret.json
DEBU[0001] Using registries.d directory /etc/containers/registries.d for sigstore configuration
DEBU[0001]  Using "default-docker" configuration
DEBU[0001]  No signature storage configuration found for registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766
DEBU[0001] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io
DEBU[0001] GET https://registry.redhat.io/v2/
DEBU[0002] Ping https://registry.redhat.io/v2/ status 401
DEBU[0002] GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?account=52173376%7Cuhc-1ECxqMHPJD8Nu4qi56CMSM44hvz&scope=repository%3Aopenshift4%2Fose-cluster-logging-operator%3Apull&service=dock
er-registry
DEBU[0002] GET https://registry.redhat.io/v2/openshift4/ose-cluster-logging-operator/tags/list




Version-Release number of selected component (if applicable):

- OCP 4.5.2 
- Cluster Logging: 4.5.0-202010090328.p0 

How reproducible:


Steps to Reproduce:
1. Install OCP4 Air-gapped - follow the documention
2. Sync redhat operators - follow the documentation
3. Install cluster logging operator

Actual results:

cluster-logging-operator pod fails to start with: Back-off pulling image "registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766"


Expected results:

cluster-logging-operator pod should start

Additional info:

My lab is available for tests, please let me know If I can support.


The cluster-logging.v4.5.0.clusterserviceversion.yaml containers the failty sha:

$ grep -A1 -B1 ose-cluster-logging-operator cluster-logging.v4.5.0.clusterserviceversion.yaml
      The Cluster Logging Operator for OKD provides a means for configuring and managing your aggregated logging stack.
    containerImage: registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766
    createdAt: 2018-08-01T08:00:00Z
--
  relatedImages:
    - name: ose-cluster-logging-operator
      image: registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766
    - name: ose-logging-curator5
--
              - name: cluster-logging-operator
                image: registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766
                imagePullPolicy: IfNotPresent

Source of the  cluster-logging.v4.5.0.clusterserviceversion.yaml  cache of "oc adm catalog build"

Comment 1 Robert Bohne 2020-10-23 13:24:21 UTC
Runs os image mirror with -v=8, it looks like that is multiarch image problem:

-v=8 hint: multi arch “forward”
I1023 14:19:56.220387  583676 mirror.go:491] Found manifest sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 with type *manifestlist.DeserializedManifestList
I1023 14:19:56.220408  583676 manifest.go:355] Including image for manifestlist.PlatformSpec{Architecture:"amd64", OS:"linux", OSVersion:"", OSFeatures:[]string(nil), Variant:"", Features:[]string(nil)} from registry.redhat.io/openshift4/ose-cluster-logging-operator
I1023 14:19:56.220430  583676 manifest.go:352] Skipping image for manifestlist.PlatformSpec{Architecture:"ppc64le", OS:"linux", OSVersion:"", OSFeatures:[]string(nil), Variant:"", Features:[]string(nil)} from registry.redhat.io/openshift4/ose-cluster-logging-operator
I1023 14:19:56.220444  583676 manifest.go:352] Skipping image for manifestlist.PlatformSpec{Architecture:"s390x", OS:"linux", OSVersion:"", OSFeatures:[]string(nil), Variant:"", Features:[]string(nil)} from registry.redhat.io/openshift4/ose-cluster-logging-operator
I1023 14:19:56.220495  583676 manifest.go:379] Filtered manifest list to new digest sha256:077ee9ffb6146e26b751eb432aabe77339dd996a9cc0aaaf26ade67eb2eb4bba:
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1371,
         "digest": "sha256:150885927c864c519664629cfb404f9dd279c7162562a370a0928b291012c6ce",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }
   ]
}

Work-a-round is to run oc image mirror or oc adm catalog mirror with --filter-by-os='.*'

I don't know who is responsible for mirror stuff - might the OLM part.

Comment 3 Robert Bohne 2020-10-26 09:39:54 UTC
--keep-manifest-list do not solve the problem, with 4.5.13 is --keep-manifest-list=false by default.


$ oc version
Client Version: 4.5.13

$ oc image mirror --help 2>&1 | grep keep-manifest
      --keep-manifest-list=false: If an image is part of a manifest list, always mirror the list even if only one image is found. The default is to mirror the specific image unless unless --filter-by-os is '.*'.

$ cat hetzner-ocp4/redhat-operators-${SERIAL}/mapping-f90c4.txt 
registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766=host.compute.local:5000/openshift4/ose-cluster-logging-operator:79807487

$ skopeo inspect --authfile ${LOCAL_SECRET_JSON} docker://${LOCAL_REGISTRY}/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 | jq '.Name'                                                                                                
FATA[0000] Error parsing image name "docker://host.compute.local:5000/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766": Error reading manifest sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 in host.compute.local:5000/openshift4/ose-cluster-logging-operator: manifest unknown: manifest unknown
                                                                 
$ oc image mirror -a ${LOCAL_SECRET_JSON} -f redhat-operators-${SERIAL}/mapping-f90c4.txt  --keep-manifest-list=true                               
host.compute.local:5000/
  openshift4/ose-cluster-logging-operator
    blobs:
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:2bd25ca124579d6fce8668ff5d4ed83866d7e7438cb561a51ddde8cc40272822 1.768KiB   
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:e0226706a58476937aea5ef240e4a71268cf00b342925a7af4b124b80197c807 5.483KiB   
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:de26e5fd8a5a0e38eaa7f357f0f0d3959280223022b6a40d0a62ec46cc863744 3.353MiB   
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:e965397e401762a7c91d85ecf0b31d8ce5b0fd433048b653786aaec19f9dfd6f 9.155MiB   
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:0c99d6c60bffe97c7d577e79d738047d711b1fc5b372c78d999d5bd08247db75 20.99MiB   
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:1323a241cc068f2816dd88f00168be73339471d6dc6eb2e6c761b63b734501b6 72.85MiB   
    manifests:
      sha256:077ee9ffb6146e26b751eb432aabe77339dd996a9cc0aaaf26ade67eb2eb4bba                                                                       
      sha256:150885927c864c519664629cfb404f9dd279c7162562a370a0928b291012c6ce                                                                       
      sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 -> 79807487                                                           
  stats: shared=0 unique=6 size=106.4MiB ratio=1.00

phase 0:
  host.compute.local:5000 openshift4/ose-cluster-logging-operator blobs=6 mounts=0 manifests=3 shared=0                                             

info: Planning completed in 1.42s
uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:de26e5fd8a5a0e38eaa7f357f0f0d3959280223022b6a40d0a62ec46cc863744 3.353MiB
uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:1323a241cc068f2816dd88f00168be73339471d6dc6eb2e6c761b63b734501b6 72.85MiB
uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:e965397e401762a7c91d85ecf0b31d8ce5b0fd433048b653786aaec19f9dfd6f 9.155MiB
uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:0c99d6c60bffe97c7d577e79d738047d711b1fc5b372c78d999d5bd08247db75 20.99MiB
sha256:150885927c864c519664629cfb404f9dd279c7162562a370a0928b291012c6ce host.compute.local:5000/openshift4/ose-cluster-logging-operator             
sha256:077ee9ffb6146e26b751eb432aabe77339dd996a9cc0aaaf26ade67eb2eb4bba host.compute.local:5000/openshift4/ose-cluster-logging-operator:79807487    
error: unable to push manifest to host.compute.local:5000/openshift4/ose-cluster-logging-operator: manifest blob unknown: blob unknown to registry  
info: Mirroring completed in 7.63s (14.6MB/s)
error: one or more errors occurred while uploading images

$ skopeo inspect --authfile ${LOCAL_SECRET_JSON} docker://${LOCAL_REGISTRY}/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 | jq '.Name'
FATA[0000] Error parsing image name "docker://host.compute.local:5000/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766": Error reading manifest sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 in host.compute.local:5000/openshift4/ose-cluster-logging-operator: manifest unknown: manifest unknown


Only add --filter-by-os=".*" solve the problem: 

$ oc image mirror -a ${LOCAL_SECRET_JSON} -f redhat-operators-${SERIAL}/mapping-f90c4.txt  --keep-manifest-list=true --filter-by-os=".*"
host.compute.local:5000/
  openshift4/ose-cluster-logging-operator
    blobs:
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:2bd25ca124579d6fce8668ff5d4ed83866d7e7438cb561a51ddde8cc40272822 1.768KiB
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:d1b57d73d9c8c74ed27f33a527e3008d7ddea4328b719df90bc74e1fadd069e9 1.784KiB
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:ea26005206bce5d3cfac2da18b76ebd27865b39c5aa9b4a2c0e78a4842bba3b6 1.793KiB
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:db58e12a850c0fc9eed05dc47557608080f6a9a11a007e9044befda75bddfc30 5.482KiB
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:e0226706a58476937aea5ef240e4a71268cf00b342925a7af4b124b80197c807 5.483KiB
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:e3f1476ea3864481132101529a327282d03ce7c48f414916b533d3a83c21871e 5.494KiB
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:de26e5fd8a5a0e38eaa7f357f0f0d3959280223022b6a40d0a62ec46cc863744 3.353MiB
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:523e1ad755e7dae4a72c92ee466af48b8a3f740a3fd1682b776519038a6675ff 3.611MiB
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:81d7c66c2a31d6196074b5f39e32a10a6edaa997d526fbab40ea2f9b11be8d8b 4.189MiB
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:e965397e401762a7c91d85ecf0b31d8ce5b0fd433048b653786aaec19f9dfd6f 9.155MiB
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:5f88f8a633957a79091938fb5fd47516253f941f63c2421098572c1badd1ee34 9.704MiB
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:54682c4eb4192cd24ce12b88009758c1bc4e73dca9a03e8480f0f1d90166022f 9.818MiB
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:ad20b01de7b73810b971e23bd0cb410a80f397fd9598002f39fc2fe7b777207f 19.14MiB
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:0c99d6c60bffe97c7d577e79d738047d711b1fc5b372c78d999d5bd08247db75 20.99MiB
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:a9a8f817082ad17ccbc857524cd5f6be7cf4a90bfce035eca05611250a8e216c 21.52MiB
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:1323a241cc068f2816dd88f00168be73339471d6dc6eb2e6c761b63b734501b6 72.85MiB
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:184b56835891110cbed643cdfd628d5b770069be584d83bfbe6fb0666c87645b 73.27MiB
      registry.redhat.io/openshift4/ose-cluster-logging-operator sha256:77a0bc0be79a60e619bff47345e7937c222a3705cff3cf9f90e17fbd5927f0cb 76.86MiB
    manifests:
      sha256:150885927c864c519664629cfb404f9dd279c7162562a370a0928b291012c6ce
      sha256:96056c0aeec9e950a03e638318b26d6e52caace77023e32d18ddbd7775ebdd10
      sha256:d40817849b0f18e2839ed2d8641950ce80538d0958630167aff81c31f2cde89c
      sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766
      sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 -> 79807487
  stats: shared=0 unique=18 size=324.5MiB ratio=1.00

phase 0:
  host.compute.local:5000 openshift4/ose-cluster-logging-operator blobs=18 mounts=0 manifests=5 shared=0

info: Planning completed in 4.38s
uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:a9a8f817082ad17ccbc857524cd5f6be7cf4a90bfce035eca05611250a8e216c 21.52MiB                       
uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:523e1ad755e7dae4a72c92ee466af48b8a3f740a3fd1682b776519038a6675ff 3.611MiB                       
uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:77a0bc0be79a60e619bff47345e7937c222a3705cff3cf9f90e17fbd5927f0cb 76.86MiB                       
uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:184b56835891110cbed643cdfd628d5b770069be584d83bfbe6fb0666c87645b 73.27MiB                       
uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:81d7c66c2a31d6196074b5f39e32a10a6edaa997d526fbab40ea2f9b11be8d8b 4.189MiB                       
uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:54682c4eb4192cd24ce12b88009758c1bc4e73dca9a03e8480f0f1d90166022f 9.818MiB                       
uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:5f88f8a633957a79091938fb5fd47516253f941f63c2421098572c1badd1ee34 9.704MiB                       
uploading: host.compute.local:5000/openshift4/ose-cluster-logging-operator sha256:ad20b01de7b73810b971e23bd0cb410a80f397fd9598002f39fc2fe7b777207f 19.14MiB                       
sha256:96056c0aeec9e950a03e638318b26d6e52caace77023e32d18ddbd7775ebdd10 host.compute.local:5000/openshift4/ose-cluster-logging-operator
sha256:150885927c864c519664629cfb404f9dd279c7162562a370a0928b291012c6ce host.compute.local:5000/openshift4/ose-cluster-logging-operator
sha256:d40817849b0f18e2839ed2d8641950ce80538d0958630167aff81c31f2cde89c host.compute.local:5000/openshift4/ose-cluster-logging-operator
sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 host.compute.local:5000/openshift4/ose-cluster-logging-operator:79807487
info: Mirroring completed in 9.35s (24.44MB/s)

$ skopeo inspect --authfile ${LOCAL_SECRET_JSON} docker://${LOCAL_REGISTRY}/openshift4/ose-cluster-logging-operator@sha256:f90c4e64328bffb136a4a76e54df674c85ae454206a6f8ff76e857a5e4825766 | jq '.Name'
"host.compute.local:5000/openshift4/ose-cluster-logging-operator"

Comment 5 Kevin Rizza 2020-10-28 17:21:09 UTC
*** Bug 1891549 has been marked as a duplicate of this bug. ***

Comment 7 Alexander Greene 2020-10-29 20:06:11 UTC
I am updating the Severity and Priority of this ticket due to the number of cases that are being reported.

Comment 12 Kevin Rizza 2020-12-01 14:43:34 UTC
*** Bug 1898853 has been marked as a duplicate of this bug. ***

Comment 13 dgangaia 2020-12-02 00:42:16 UTC
Hi, 

 We do have a customer hitting the same issue, However none of the workaround suggest works for customer.Please let us know if there is any other workaround.

Thanks,
Dixit

Comment 14 Robert Bohne 2020-12-02 07:07:41 UTC
Created attachment 1735497 [details]
Bug1890951-tests-with-ocp4.6.6.details.log

The problem still exists with 4.6.6.

Here my simplified summary, all the details attached.

During the sync with
oc adm catalog mirror \
  ${LOCAL_REGISTRY}/redhat/redhat-operator-index:v${OCP_RELEASE%\.*} \
  ${LOCAL_REGISTRY} \
it sync the manifest list, BUT it removes the not needed architectures. This breaks the digest of the manifest list by design! But the digest of the full manifest list is used for the operator deployment:

      image: >-
        registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:540b0d087c5e5529bab555030310478630249a9339a4a4c3fac6d0d7037d5eac

Here the details of the debug sync log:

I1202 06:40:03.920447 2096345 mirror.go:513] Found manifest sha256:540b0d087c5e5529bab555030310478630249a9339a4a4c3fac6d0d7037d5eac with type *manifestlist.DeserializedManifestList
I1202 06:40:03.920465 2096345 manifest.go:373] Including image sha256:5518705b8d98dcb6dfbfb2ba5b375ec052944cdea8667f9f26cbe5a713780118 for manifestlist.PlatformSpec{Architecture:"amd64", OS:"linux", OSVersion:"", OSFeatures:[]string(nil), Variant:"", Features:[]string(nil)} from registry.redhat.io/openshift4/ose-cluster-logging-operator
I1202 06:40:03.920481 2096345 manifest.go:370] Skipping image sha256:5b3fa37a4fba08eddc92a13ed45e469f6d3d089f4580fff35646e507f15620bd for manifestlist.PlatformSpec{Architecture:"ppc64le", OS:"linux", OSVersion:"", OSFeatures:[]string(nil), Variant:"", Features:[]string(nil)} from registry.redhat.io/openshift4/ose-cluster-logging-operator
I1202 06:40:03.920491 2096345 manifest.go:370] Skipping image sha256:7ffe5e80d9ffcfc2d6fd47add4bc110ccec8ae4a8232f6bdd25bea30e9202656 for manifestlist.PlatformSpec{Architecture:"s390x", OS:"linux", OSVersion:"", OSFeatures:[]string(nil), Variant:"", Features:[]string(nil)} from registry.redhat.io/openshift4/ose-cluster-logging-operator
I1202 06:40:03.920525 2096345 manifest.go:397] Filtered manifest list to new digest sha256:236052852d9a5bca319e7734c041d7cd18679f21dbca4dbce87aee615b415074:
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1372,
         "digest": "sha256:5518705b8d98dcb6dfbfb2ba5b375ec052944cdea8667f9f26cbe5a713780118",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }
   ]

Ergo sha256:540b0d087c5e5529bab555030310478630249a9339a4a4c3fac6d0d7037d5eac is not available anymore.

That looks like a design problem, from my understanding you can not push a full manifest list unless all other architectures digests are available at the target registry. As ecordell mention [1]

In case all the multiarch manifests stuff is new to you, I wrote a blogpost about it [2] because it was also new to me :-)


The work-a-round to add `--filter-by-os='.*'` still works for me.


[1] https://github.com/openshift/oc/pull/642#issuecomment-736758745
[2] https://www.opensourcerers.org/2020/11/16/container-images-multi-architecture-manifests-ids-digests-whats-behind/

Comment 15 dgangaia 2020-12-03 05:28:19 UTC
Thank Robert, Had a call with customer today and we were able to resolve the issue using `--filter-by-os='.*'` , basically new catalog image built using "oc adm catalog build" was not getting picked during catalogsoure deployment since an old image existed on the node and it was not pulling the new image. After "oc adm prune images" and removing all older catalogsource we were able to deploy a new catalogsource and then cluster logging operator installed successfully.

Regards,
Dixit

Comment 16 Maciej Szulik 2020-12-03 10:30:36 UTC
*** Bug 1903230 has been marked as a duplicate of this bug. ***

Comment 17 Sally 2020-12-05 00:11:24 UTC
Also, there's an open PR to address confusion over --filter-by-os=.* vs --keep-manifest-list=true, once that PR merges I'll update here since I'm not sure what the decision will be regarding these flags, PR is under review. (openshift oc pull 642)

Comment 21 Alex Dellapenta 2020-12-17 16:06:16 UTC
Merged OCP docs PR for the --filter-by-os='.*' workaround: https://github.com/openshift/openshift-docs/pull/27855

Updated doc has been published (step 2, callout 4): https://docs.openshift.com/container-platform/4.5/operators/admin/olm-restricted-networks.html#olm-restricted-networks-operatorhub_olm-restricted-networks

Same for 4.6.

Comment 24 Andreas Karis 2020-12-23 10:14:42 UTC
Something is not right with the --filter-by-os='.*' filter in OpenShift 4.6:

https://docs.openshift.com/container-platform/4.6/operators/admin/olm-restricted-networks.html#olm-mirror-catalog_olm-restricted-networks

~~~
oc adm catalog mirror \
    <index_image> \
    <mirror_registry>:<port> \
    [-a ${REG_CREDS}] \
    [--insecure] \
    --filter-by-os='.*' \
    [--manifests-only] 
~~~

The doc clearly states:
~~~
 	This flag is currently required due to a known issue with multiple architecture support. If unset or set to any value other than .*, filtering out different architectures changes the digest of the manifest list, also known as a "multi-arch image", which causes deployments of those images and Operators on disconnected clusters to fail. For more information, see BZ#1890951.
~~~

When I run this, I get:
~~~
[root@openshift-jumpserver-0 ~]# oc adm catalog mirror registry.redhat.io/redhat/redhat-operator-index:v4.6 192.168.123.1:5000 -a ${REG_CREDS} --filter-by-os='.*'
error: the image is a manifest list and contains multiple images - use --filter-by-os to select from:

  OS            DIGEST
  linux/amd64   sha256:7d1e513ffad86c02189de25479aebef62ca4cef7d588cb3910ae1791c1537d78
  linux/ppc64le sha256:14ca1b701ba1d95b8de22100738325469442a8aaf6d655ddfe4ca511b7fb07af
  linux/s390x   sha256:ccac6405ca2658c03c6f262fd23ef7c97584d2e1f676eaba31943110b99fbff7

[root@openshift-jumpserver-0 ~]# oc adm catalog mirror registry.redhat.io/redhat/redhat-operator-index:v4.6 192.168.123.1:5000 -a ${REG_CREDS} --filter-by-os=.*
error: the image is a manifest list and contains multiple images - use --filter-by-os to select from:

  OS            DIGEST
  linux/amd64   sha256:7d1e513ffad86c02189de25479aebef62ca4cef7d588cb3910ae1791c1537d78
  linux/ppc64le sha256:14ca1b701ba1d95b8de22100738325469442a8aaf6d655ddfe4ca511b7fb07af
  linux/s390x   sha256:ccac6405ca2658c03c6f262fd23ef7c97584d2e1f676eaba31943110b99fbff7
~~~

Comment 25 Andreas Karis 2020-12-23 10:17:17 UTC
That's with:
[root@openshift-jumpserver-0 ~]# oc version
Client Version: 4.6.1
Server Version: 4.6.1

Comment 26 Andreas Karis 2020-12-23 10:20:05 UTC
I updated the client to 4.6.8 and still see the same:

[root@openshift-jumpserver-0 ~]# oc adm catalog mirror registry.redhat.io/redhat/redhat-operator-index:v4.6 192.168.123.1:5000 -a ${REG_CREDS} --filter-by-os=.*
error: the image is a manifest list and contains multiple images - use --filter-by-os to select from:

  OS            DIGEST
  linux/amd64   sha256:7d1e513ffad86c02189de25479aebef62ca4cef7d588cb3910ae1791c1537d78
  linux/ppc64le sha256:14ca1b701ba1d95b8de22100738325469442a8aaf6d655ddfe4ca511b7fb07af
  linux/s390x   sha256:ccac6405ca2658c03c6f262fd23ef7c97584d2e1f676eaba31943110b99fbff7

[root@openshift-jumpserver-0 ~]# oc adm catalog mirror registry.redhat.io/redhat/redhat-operator-index:v4.6 192.168.123.1:5000 -a ${REG_CREDS} --filter-by-os=".*"
error: the image is a manifest list and contains multiple images - use --filter-by-os to select from:

  OS            DIGEST
  linux/amd64   sha256:7d1e513ffad86c02189de25479aebef62ca4cef7d588cb3910ae1791c1537d78
  linux/ppc64le sha256:14ca1b701ba1d95b8de22100738325469442a8aaf6d655ddfe4ca511b7fb07af
  linux/s390x   sha256:ccac6405ca2658c03c6f262fd23ef7c97584d2e1f676eaba31943110b99fbff7

[root@openshift-jumpserver-0 ~]# oc version
Client Version: 4.6.8
Server Version: 4.6.1
Kubernetes Version: v1.19.0+d59ce34

Comment 27 yeneryumlu 2021-01-05 06:26:34 UTC
I am using OCP 4.6.9 and the latest version of oc cli.
I am getting the same error message like Andreas with --filter-by-os='.*' parameter.

[core@bastiontest olm]$ oc adm catalog mirror registry.redhat.io/redhat/redhat-operator-index:v4.6 scorenexus.isbank:9443/olmr -a olm_pull-secret --filter-by-os='.*' --manifests-only
error: the image is a manifest list and contains multiple images - use --filter-by-os to select from:

  OS            DIGEST
  linux/amd64   sha256:1b13f5d739219cf4aaf2578e1fe455bb6e1ac58c545cacfe2efd38c721e6ff85
  linux/ppc64le sha256:b5954897d5487b956097a094fbf444dd4d84dfa056206286adc2cd49b69ebf3d
  linux/s390x   sha256:8804c0d52b9ff5fab9357b89415e08f0462d1db15306f579b0e09dc938864e27

[core@bastiontest olm]$ oc version
Client Version: openshift-clients-4.6.0-202006250705.p0-168-g02c110006
Server Version: 4.6.9
Kubernetes Version: v1.19.0+7070803

Comment 32 Stu Lipshires 2021-02-08 12:00:52 UTC
I am getting this on OC 4.6.15 client. Please advise.


[root@bastion ~]# oc adm catalog mirror registry.redhat.io/redhat/redhat-operator-index:v4.6  bastion.airgap1.cdastu.com:5000 -a ${REG_CREDS} --filter-by-os=".*" --insecure
error: the image is a manifest list and contains multiple images - use --filter-by-os to select from:

  OS            DIGEST
  linux/amd64   sha256:804d289905c20889e50aec6728a85020d8f30ca46f88d84e34683cef2918eb18
  linux/ppc64le sha256:78eee018c0c3db14201ccbef4084836564080c9258a97427ca24b018ba9823e0
  linux/s390x   sha256:805ceb30b17ccd942178626480df1cb60f99706792b0f79ecd88c7aedffae194

[root@bastion ~]# oc version
Client Version: 4.6.15
Server Version: 4.6.15
Kubernetes Version: v1.19.0+1833054

Comment 33 Kevin Rizza 2021-02-23 13:41:48 UTC
Hi Stu,

You should just no longer set filter-by-os in the latest version of oc. By default, it will select an arch option if you don't set it. The flag is still there for backwards compatibility reasons, but for the latest images selecting .* will never work because the index image itself has multiple architectures and one has to be picked to pull out the catalog contents from it.

Comment 34 Stu Lipshires 2021-02-23 14:28:13 UTC
OK, that worked. I assume that there will be a Doc change on this?


Note You need to log in before you can comment on or make changes to this bug.