Created attachment 1713144 [details] output of oc adm catalog mirror Description of problem: I have a disconnected OCP 4.6.fc.2 cluster and I'm mirroring operators from registry-proxy.engineering.redhat.com/rh-osbs/iib-pub-pending:v4.5. Mirroring works fine but installation of operator fails. Version-Release number of selected component (if applicable): OCP 4.6.fc.2 How reproducible: Always Steps to Reproduce: 1. install disconnected cluster with mirror registry following https://docs.openshift.com/container-platform/4.5/installing/install_config/installing-restricted-networks-preparations.html 2. disable default sources: oc patch operatorhub.config.openshift.io/cluster -p='{"spec":{"disableAllDefaultSources":true}}' --type=merge 3. on a host with the mirror registry a) add 'registry-proxy.engineering.redhat.com','registry.stage.redhat.io' to insecure registries in /etc/containers/registries.conf b) run podman login for registry.stage.redhat.io, registry.redhat.io and mirror registry and copy auth.json to current dir c) mirror catalog with --manifests-only: oc adm catalog mirror registry-proxy.engineering.redhat.com/rh-osbs/iib-pub-pending:v4.5 `hostname`:55555/olm -a auth.json --insecure --manifests-only d) get only kiali and jaeger from mapping.txt: cat iib-pub-pending-manifests/mapping.txt | grep -E 'kiali|jaeger' > map.txt e) mirror images from map.txt: oc image mirror -a auth.json --insecure -f map.txt --filter-by-os=/* f) previous fails because of registry.redhat.io/openshift-service-mesh/kiali-rhel7 manifest sha256:29acbaf2c5c6909b0a9596060a40a2f304047c361422442fccc674a535fcee6a does not exist in registry.redhat.io -> change it to following in map.txt: registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-kiali-rhel7@sha256:29acbaf2c5c6909b0a9596060a40a2f304047c361422442fccc674a535fcee6a=bastion.fbr-46-2-disc.maistra.upshift.redhat.com:55555/olm/openshift-service-mesh-kiali-rhel7:9258190e g) run step e) again which should be successful h) mirror IIB itself: podman inspect --format='{{index .RepoDigests 0}}' registry-proxy.engineering.redhat.com/rh-osbs/iib-pub-pending:v4.5 oc image mirror -a auth.json --insecure registry-proxy.engineering.redhat.com/rh-osbs/iib-pub-pending@sha256:3ea4b0a44c3422219edb7ed5817385ff221be58ffa09ac2ddfc6c331281a42b1 bastion.fbr-46-2-disc.maistra.upshift.redhat.com:55555/olm/my-iib:mytag i) oc apply -f iib-pub-pending-manifests/imageContentSourcePolicy.yaml j) wait for the cluster to reconcile k) create new catalog source with following content: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: my-index-catalog namespace: openshift-marketplace spec: sourceType: grpc image: bastion.fbr-46-2-disc.maistra.upshift.redhat.com:55555/olm/my-iib:mytag l) go to UI console and install kiali and jaeger operators with all values set to default Actual results: Installation never finishes but both operators are marked as installed on Operator Hub page. Operators are not visible on Installed operators page. There are no pending pods or errors on Overview page. There is only one event which seems to be related - found more than one head for channel. No pods are running in redhat-operators NS. marketplace-operator and my-index-catalog pods are running in openshift-marketplace NS. Expected results: Operators are installed. Additional info: Attached output of oc adm catalog mirror command, output of oc image mirror command. Content of imageContentSourcePolicy.yaml and edited mapping.txt.
Created attachment 1713145 [details] imageContentSourcePolicy.yaml
Created attachment 1713146 [details] output of oc image mirror command
Created attachment 1713147 [details] edited mapping.txt
I can see following on jaeger-product subscription page: Install Plan None Upgrade Status Unknown failure 0 installed0 installing
Found easier repro steps which are showing the same error. OCP 4.6.fc.4 1) Install OCP 4.6.fc.4 (standard cluster, no need for disconnected) 2) disable default catalog sources: oc patch operatorhub.config.openshift.io/cluster -p='{"spec":{"disableAllDefaultSources":true}}' --type=merge 3) add insecure registries: oc patch --type=merge --patch='{"spec":{"registrySources":{"insecureRegistries":[ "registry.stage.redhat.io","registry-proxy-stage.engineering.redhat.com"]}}}' image.config.openshift.io/cluster 4) create following catalog source: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: my-index-catalog-stage namespace: openshift-marketplace spec: sourceType: grpc image: registry.stage.redhat.io/redhat/redhat-operator-index:v4.6 5) prepare imageContentSourcePolicy: oc adm catalog mirror registry.stage.redhat.io/redhat/redhat-operator-index:v4.6 registry.stage.redhat.io --insecure --manifests-only -a /run/user/1000/containers/auth.json 6) oc apply -f redhat-operator-index-manifests/imageContentSourcePolicy.yaml 7) wait for cluster to reconcile 8) install kiali operator with default values Result: Stuck on Installing... Kiali operator marked as installed on Operator hub page but not visible on Installed operators page There is no install plan created. There are no errors visible only following event: openshift-operators found more than one head for channel This issue is blocking for us to test kiali operator using registry.stage.redhat.io/redhat/redhat-operator-index:v4.6 on OCP 4.6 Note that the same steps just using registry.stage.redhat.io/redhat/redhat-operator-index:v4.5 worked fine on OCP 4.5
Forgot to mention one step in reprosteps in previous comment - it's necessary to create a secret to be able to pull from registry.stage.redhat.io
It looks like there is some problem with kiali-operator.v1.12.14 which is the latest version available in registry.stage.redhat.io/redhat/redhat-operator-index:v4.6. I manually added v1.12.15 via: opm index add --bundles registry-proxy.engineering.redhat.com/rh-osbs/openshift-service-mesh-kiali-operator-metadata@sha256:76be72987f48e53c1753789e07165a7571d2f0dbc111cc9d3d0b573ecd811fc2 --from-index registry.stage.redhat.io/redhat/redhat-operator-index:v4.6 --tag quay.io/fbrychta/test:1.0.0 And installation from quay.io/fbrychta/test:1.0.0 worked fine.
So the registry.stage.redhat.io/redhat/redhat-operator-index:v4.6 now contains kiali-operator.v1.12.15 but the issue is still visible. When using registry.stage.redhat.io/redhat/redhat-operator-index:v4.6 on OCP 4.5 it works fine.
> found more than one head for channel This message indicates that you're hitting a bug fixed in the linked BZ. Testing on a nightly that includes that fix should resolve the problem. (note - there are a couple of causes for this message, but the fc.4 release you used has all but the one in the linked BZ fixed).
*** This bug has been marked as a duplicate of bug 1869441 ***
I was able to install both kiali and jaeger in 4.6.0-0.nightly-2020-09-16-000734 from registry.stage.redhat.io/redhat/redhat-operator-index:v4.6