Created attachment 1713273 [details] imageContentSourcePolicy Description of problem: I have a disconnected OCP 4.6.fc.2 cluster and I'm mirroring operators from our internal IIB quay.io/maistra/iib:latest-qe. Mirroring works fine but installation of operator fails with: Failed to pull image "registry-proxy.engineering.redhat.com/rh-osbs/distributed-tracing-jaeger-rhel7-operator-metadata@sha256:66ee9862da3be7da50ae894e5f5cef9946543adc612a0456541494f15b56412b": rpc error: code = Unknown desc = (Mirrors also failed: [bastion.fbr-46-disc.maistra.upshift.redhat.com:55555/olm/rh-osbs-distributed-tracing-jaeger-rhel7-operator-metadata@sha256:66ee9862da3be7da50ae894e5f5cef9946543adc612a0456541494f15b56412b: Error reading manifest sha256:66ee9862da3be7da50ae894e5f5cef9946543adc612a0456541494f15b56412b in bastion.fbr-46-disc.maistra.upshift.redhat.com:55555/olm/rh-osbs-distributed-tracing-jaeger-rhel7-operator-metadata: manifest unknown: manifest unknown]): registry-proxy.engineering.redhat.com/rh-osbs/distributed-tracing-jaeger-rhel7-operator-metadata@sha256:66ee9862da3be7da50ae894e5f5cef9946543adc612a0456541494f15b56412b: error pinging docker registry registry-proxy.engineering.redhat.com: Get "https://registry-proxy.engineering.redhat.com/v2/": dial tcp 10.0.19.22:443: i/o timeout Version-Release number of selected component (if applicable): OCP 4.6.fc.2 How reproducible: Always Steps to Reproduce: 1. install disconnected cluster with mirror registry following https://docs.openshift.com/container-platform/4.5/installing/install_config/installing-restricted-networks-preparations.html 2. disable default sources: oc patch operatorhub.config.openshift.io/cluster -p='{"spec":{"disableAllDefaultSources":true}}' --type=merge 3. on a host with the mirror registry a) podman login to all required registires (quay.io,registry.redhat.io, mirror registry) copy auth.json to current dir b) mirror the catalog (see additional info on iib bellow): oc adm catalog mirror quay.io/maistra/iib:latest-qe `hostname`:55555/olm -a auth.json --insecure c) mirror IIB itself: podman inspect --format='{{index .RepoDigests 0}}' registry-quay.io/maistra/iib:latest-qe oc image mirror quay.io/maistra/iib@sha256:dc43ea4c510742b6888ceca954be659449b70cee5deed2d823c3525b5d38ff27 `hostname`:55555/olm/iib-my:mytag -a auth.json --insecure d) oc apply -f iib-manifests/imageContentSourcePolicy.yaml e) wait for the cluster to reconcile f) create new catalog source with following content: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: my-index-catalog namespace: openshift-marketplace spec: sourceType: grpc image: bastion.fbr-46-disc.maistra.upshift.redhat.com:55555/olm/iib-my:mytag g) go to UI console and install jaeger operator with all values set to default Actual results: Failed to pull image "registry-proxy.engineering.redhat.com/rh-osbs/distributed-tracing-jaeger-rhel7-operator-metadata@sha256:66ee9862da3be7da50ae894e5f5cef9946543adc612a0456541494f15b56412b": rpc error: code = Unknown desc = (Mirrors also failed: [bastion.fbr-46-disc.maistra.upshift.redhat.com:55555/olm/rh-osbs-distributed-tracing-jaeger-rhel7-operator-metadata@sha256:66ee9862da3be7da50ae894e5f5cef9946543adc612a0456541494f15b56412b: Error reading manifest sha256:66ee9862da3be7da50ae894e5f5cef9946543adc612a0456541494f15b56412b in bastion.fbr-46-disc.maistra.upshift.redhat.com:55555/olm/rh-osbs-distributed-tracing-jaeger-rhel7-operator-metadata: manifest unknown: manifest unknown]): registry-proxy.engineering.redhat.com/rh-osbs/distributed-tracing-jaeger-rhel7-operator-metadata@sha256:66ee9862da3be7da50ae894e5f5cef9946543adc612a0456541494f15b56412b: error pinging docker registry registry-proxy.engineering.redhat.com: Get "https://registry-proxy.engineering.redhat.com/v2/": dial tcp 10.0.19.22:443: i/o timeout Expected results: Operator should be installed Additional info: Attached imageContentSourcePolicy.yaml, mapping.txt, output of mirroring command. Note that imageContentSourcePolicy.yaml contains: - mirrors: - bastion.fbr-46-disc.maistra.upshift.redhat.com:55555/olm/rh-osbs-distributed-tracing-jaeger-rhel7-operator-metadata source: registry-proxy.engineering.redhat.com/rh-osbs/distributed-tracing-jaeger-rhel7-operator-metadata and registry-proxy.engineering.redhat.com/rh-osbs/distributed-tracing-jaeger-rhel7-operator-metadata@sha256:66ee9862da3be7da50ae894e5f5cef9946543adc612a0456541494f15b56412b is mirrored correclty: oc image mirror -a auth.json --insecure registry-proxy.engineering.redhat.com/rh-osbs/distributed-tracing-jaeger-rhel7-operator-metadata@sha256:66ee9862da3be7da50ae894e5f5cef9946543adc612a0456541494f15b56412b=bastion.fbr-46-disc.maistra.upshift.redhat.com:55555/olm/rh-osbs-distributed-tracing-jaeger-rhel7-operator-metadata:807bd786 bastion.fbr-46-disc.maistra.upshift.redhat.com:55555/ olm/rh-osbs-distributed-tracing-jaeger-rhel7-operator-metadata manifests: sha256:d18b4091d119023585b25f1d91eedf85e1f3cfd8b82e97f4327b669a6e5330ae -> 807bd786 stats: shared=0 unique=0 size=0B phase 0: bastion.fbr-46-disc.maistra.upshift.redhat.com:55555 olm/rh-osbs-distributed-tracing-jaeger-rhel7-operator-metadata blobs=0 mounts=0 manifests=1 shared=0 info: Planning completed in 370ms sha256:d18b4091d119023585b25f1d91eedf85e1f3cfd8b82e97f4327b669a6e5330ae bastion.fbr-46-disc.maistra.upshift.redhat.com:55555/olm/rh-osbs-distributed-tracing-jaeger-rhel7-operator-metadata:807bd786 info: Mirroring completed in 10ms (0B/s) To reproduce our private iib: quay.io/maistra/iib:latest-qe contains following images: oc image extract quay.io/maistra/iib:latest-qe --file=/database/index.db [fbrychta@worklaptop tmp]$ sqlite3 index.db "select name,bundlepath from operatorbundle" elasticsearch-operator.4.6.0-202006251541.p0|registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-elasticsearch-operator-bundle@sha256:1bd38e4cf8251968894ec9dbc7184a36153934435b115fa2763762c71eb851e3 jaeger-operator.v1.13.2-1|registry-proxy.engineering.redhat.com/rh-osbs/distributed-tracing-jaeger-rhel7-operator-metadata@sha256:9470291681d45a761e70b7dc8d03eb2997f0b527bbbe69dbb306e85b35b73387 jaeger-operator.v1.17.6|registry-proxy.engineering.redhat.com/rh-osbs/distributed-tracing-jaeger-rhel7-operator-metadata@sha256:66ee9862da3be7da50ae894e5f5cef9946543adc612a0456541494f15b56412b kiali-operator.v1.0.10|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-kiali-operator-metadata@sha256:f753d11a53391fe91119743dbd7116c038c933d655cc413c921623f3eb686a9b kiali-operator.v1.0.11|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-kiali-operator-metadata@sha256:9fda7ae9f73e232d713e817a6774df18a3a44f27948a602d3cbf2e6e7445eb3c kiali-operator.v1.0.12|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-kiali-operator-metadata@sha256:6ef78a0aaf9dadc3df4210ca1518eea4e1816cc0c0948f25503cbe15b027ca6e kiali-operator.v1.0.5|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-kiali-operator-metadata@sha256:a5520794b6800b7ea66c245569612d1d87979dde6e38e9f702b3dc9fd469e4db kiali-operator.v1.0.6|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-kiali-operator-metadata@sha256:a1de1149bd3368fe08424ea9ead352f136bd43a77ad348c6ab98fcfcd2a33fa7 kiali-operator.v1.0.7|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-kiali-operator-metadata@sha256:1ac34515d67c7b6434d6a40f7d2550f6286105213e70f3aa40780f76963e1f70 kiali-operator.v1.0.8|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-kiali-operator-metadata@sha256:d39eda53347b5a930cc8a5859440a1c6af63d2273153caf97cceeb3e44bccabd kiali-operator.v1.0.9|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-kiali-operator-metadata@sha256:a9dd2853998648b6acbdc12a480e51cb0f6d412c0773999fdf5fc7a5ed7417cf kiali-operator.v1.12.11|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-kiali-operator-metadata@sha256:0c823fd9cab5e3e77419d0691a073d75d65d547ca7b45db83673d765b2ebace5 kiali-operator.v1.12.12|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-kiali-operator-metadata@sha256:a846964996462276890873fffeadacf29b5034cdbfb4d670a247984f908dec68 kiali-operator.v1.12.13|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-kiali-operator-metadata@sha256:cccf2f93cc99ba4ecbb27a5acd13f54864b9dea45716c8a7006740a4d696f475 kiali-operator.v1.12.14|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-kiali-operator-metadata@sha256:12ccc7df8b8e13c2234091fde357ff6d09cd3cd56827a4d255f3dde7af274168 kiali-operator.v1.12.15|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-kiali-operator-metadata@sha256:76be72987f48e53c1753789e07165a7571d2f0dbc111cc9d3d0b573ecd811fc2 kiali-operator.v1.12.7|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-kiali-operator-metadata@sha256:43d02809090725e3f2bb425f130cccc97758a44f7c1494fd39a82203c0a9ff2a servicemeshoperator.v1.0.0|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:9171f452f74351a2b53e53a41d29f4dcf01faa95cd9378180871e2ef377c45e8 servicemeshoperator.v1.0.1|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:59e7dd5a49d4fa8c929e88da890a19235bba95a1b73d9cd0d8b521b15073eb22 servicemeshoperator.v1.0.10|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:fec2bc8c51acf3f7f6fa875346e42ca01f43ef9337fcfbb0623a1f42b120b691 servicemeshoperator.v1.0.2|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:52b12310bd6caecf883bd33602a3845dc31d080b12b0fcfe85472169bc4cbf5b servicemeshoperator.v1.0.3|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:6d6578bfdb3842d8bf20b17b809278cca3b737d0c5b7c07a83222a5e39f93d28 servicemeshoperator.v1.0.4|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:ff952f5ec7c913fdc36f92badafefce472e1f4dbf6e7d747a07581bc4c025ade servicemeshoperator.v1.0.5|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:5190bcaef8b20b01139eaecaefc128c705fee7007ae45e069741b247e82ca24e servicemeshoperator.v1.0.6|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:59fa854bda4ae92858db19eb4935e0fff77c1435917417eb79d128ce06cf61bd servicemeshoperator.v1.0.7|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:4a7818039db7c0ac020565d7c5b5cb3e72cbce16c6ea6c84f1d636373c8a2047 servicemeshoperator.v1.0.8|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:2b005ce59995e4e72d514aa6742ac9b414da5b3dc3deaba1a695ecb00618b188 servicemeshoperator.v1.0.9|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:b53796530ec3400e26feeaa444973b59d55fb8db947ac12f3faf167633b47891 servicemeshoperator.v1.1.0|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:b97b63a9574e10e5c2a5d813207311319a408835d083edfa69aa7168476d5d91 servicemeshoperator.v1.1.1|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:ad8cd6a01308e2a863ddd9c4ebd32f5df29f387f13b574d2f73960b3b474121c servicemeshoperator.v1.1.2|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:ba7bae85fe599fe5b666b0f3bb73c7619ace5b8d5190b0678813cd2782125c2c servicemeshoperator.v1.1.2.2|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:df2024f98573b2b7d26dc9569cd6fe372891bf09a7a22718b4d659f426436a8c servicemeshoperator.v1.1.2.3|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:910875e1fd5696139ab20fb979b08f22ab38fbcc4b8303fa14955e8dbe45d164 servicemeshoperator.v1.1.3|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:1bb40983a8e60152d0b8362e45388074734113bbe2b641c54f9efa34e1232222 servicemeshoperator.v1.1.4|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:6fbd0076e55c46a52a0f803c0d993c5a61fd1791f9e5ddbb24daea8e0c3ad591 servicemeshoperator.v1.1.4.2|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:21ff5632e04226653a5c3dd828b8b4a728734f29d25cb0eccb9219bea5964a72 servicemeshoperator.v1.1.5|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:c0a352422a0205799446d1acfa419cf7f87ee608dd4be37c1f89fcc6669c05f1 servicemeshoperator.v1.1.5.2|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:c1695d47fd71b0df40c52b083802b2236aae29a8afadfd95c92af7ab7d9f3ddc servicemeshoperator.v1.1.6|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:728c390d62212d1f64cd2d231a0cd977910b59283627a1cc148c74d82ab83747 servicemeshoperator.v1.1.7|registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-istio-rhel8-operator-metadata@sha256:ff8a71875639ea9de2ecff41326ed72ecdbf3cbd9ed12f00303880e82d8232db
Created attachment 1713274 [details] output of oc adm catalog mirror
Created attachment 1713275 [details] mapping.txt
This is hapenning also in an AWS/GCP cluster. It seems the catalog doesn't mirror correctly what is on iib
We hit this problem again on OCP 4.5.7 with ose-elasticsearch-operator. The image was mirrored correctly with following in mapping.txt: registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:b1463becf264e75e383d8cc33eaa407002499583b40a90eed8b071b627215ed3=bastion.ocp-jqe4-disconnected.maistra.upshift.redhat.com:55555/openshift4/ose-elasticsearch-operator:2f060a82 Following in imageContentSourcePolicy.yaml: - mirrors: - bastion.ocp-jqe4-disconnected.maistra.upshift.redhat.com:55555/openshift4/ose-elasticsearch-operator source: registry.redhat.io/openshift4/ose-elasticsearch-operator But installation of ES operators still fails with: Failed to pull image "registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:b1463becf264e75e383d8cc33eaa407002499583b40a90eed8b071b627215ed3": rpc error: code = Unknown desc = (Mirrors also failed: [bastion.ocp-jqe4-disconnected.maistra.upshift.redhat.com:55555/openshift4/ose-elasticsearch-operator@sha256:b1463becf264e75e383d8cc33eaa407002499583b40a90eed8b071b627215ed3: Error reading manifest sha256:b1463becf264e75e383d8cc33eaa407002499583b40a90eed8b071b627215ed3 in bastion.ocp-jqe4-disconnected.maistra.upshift.redhat.com:55555/openshift4/ose-elasticsearch-operator: manifest unknown: manifest unknown]): registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:b1463becf264e75e383d8cc33eaa407002499583b40a90eed8b071b627215ed3: error pinging docker registry registry.redhat.io: Get https://registry.redhat.io/v2/: dial tcp 104.113.9.148:443: i/o timeout
Sending to Sally, she's dealing with ICSP topic from our side.
ICSP topic is being deferred until 4.7.
Does this mean OCP 4.6 will not support Disconnected installations?
(In reply to kconner from comment #11) > Does this mean OCP 4.6 will not support Disconnected installations? The support will similar to the current so not 100%, we've mad an assessment on it and it seems too risky to land this feature now and we haven't completed full review, yet.
Digest not being preserved has nothing to do with the ICSP feature. Generally digest not being preserved is an incorrect mirror call. Sally, this may be the filter-by-os confusion (stripping out manifest lists if they are in use) or it could be a registry that does not support schema v2.
@Clayton this is an approach we use successfully with the older operator metadata format, with the same registry, so it's likely to be the mirroring which is failing. This seems to be confirmed by some of the additional comments.
hi Filip, sorry just am digging into this now.. yes, this isn't the case of `oc` not preserving digests, just that the images are multi-arch so whatever is mirrored is a single manifest from a manifest list unless you specify to include all with the --filter-by-os=.* Please run this command w/ the steps listed in the bz description, and report back! $ oc image mirror quay.io/maistra/iib@sha256:dc43ea4c510742b6888ceca954be659449b70cee5deed2d823c3525b5d38ff27 `hostname`:55555/olm/iib-my:mytag -a auth.json --insecure --filter-by-os=.*
Hi Sally, thank you very much. Adding --filter-by-os=.* really seems to solve the problem. I verified that using different scenario which was showing the same type of error so I still need to verify the scenario from this BZ to be 100% sure. I will do that tomorrow.
I'm closing this as not a bug. The cause of the problem was missing --filter-by-os=.* parameter in step 3.b from the original description. Thank you Sally and Miloslav for your help. One thing to consider. We hit this issue because we were using existing mirroring scripts which were working fine without --filter-by-os=.* before the multi-arch images were introduced. Customers with existing mirroring scripts which are not using --filter-by-os=.* will hit the same issue. Would it make sense to change the default behaviour in a way that the oc adm catalog mirror command would continue to work without --filter-by-os=.* even for multi-arch images?
@Filip, I was thinking the same, perhaps we can always include the manifestlist when mirroring. I'll discuss with the team, thank you!