Created attachment 1712205 [details] Catalog Operator logs Description ------------ On an OCP 4.6 cluster, I'm trying to install CNV 2.5 from registry-proxy.engineering.redhat.com/rh-osbs/iib:3555 (ImageContentSourcePolicy is properly configured). After creating the Subscription to the kubevirt-hyperconverged package, I'm expecting for the CSV kubevirt-hyperconverged-operator.v2.5.0 to be created but it never happens (the InstallPlan is not created either). Logs ---- Logs from the catalog-operator have warnings about constraints not satisfiable during resolution. > kubectl -n openshift-operator-lifecycle-manager logs deployment/catalog-operator > > E0821 17:20:28.884468 1 queueinformer_operator.go:290] sync "openshift-cnv" failed: constraints not satisfiable: kubevirt-hyperconverged requires at least one of hco-catalogsource/openshift-marketplace/2.5/kubevirt-hyperconverged-operator.v2.5.0, kubevirt-hyperconverged is mandatory, gvkunique/cdi.kubevirt.io/v1alpha1/CDI permits at most 1 of hco-catalogsource/openshift-marketplace/2.4/kubevirt-hyperconverged-operator.v2.4.0, hco-catalogsource/openshift-marketplace/2.5/kubevirt-hyperconverged-operator.v2.5.0, hco-catalogsource/openshift-marketplace/2.5/kubevirt-hyperconverged-operator.v2.5.0 requires at least one of hco-catalogsource/openshift-marketplace/2.4/kubevirt-hyperconverged-operator.v2.4.0 > I0821 17:20:28.884876 1 event.go:278] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"openshift-cnv", UID:"fb4aacfb-f672-4e54-9298-c64c61c3894b", APIVersion:"v1", ResourceVersion:"105890", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' constraints not satisfiable: kubevirt-hyperconverged requires at least one of hco-catalogsource/openshift-marketplace/2.5/kubevirt-hyperconverged-operator.v2.5.0, kubevirt-hyperconverged is mandatory, gvkunique/cdi.kubevirt.io/v1alpha1/CDI permits at most 1 of hco-catalogsource/openshift-marketplace/2.4/kubevirt-hyperconverged-operator.v2.4.0, hco-catalogsource/openshift-marketplace/2.5/kubevirt-hyperconverged-operator.v2.5.0, hco-catalogsource/openshift-marketplace/2.5/kubevirt-hyperconverged-operator.v2.5.0 requires at least one of hco-catalogsource/openshift-marketplace/2.4/kubevirt-hyperconverged-operator.v2.4.0 and so on. Full logs in attachement. Versions -------- - OCP: v4.6.0-fc.0 - hco-bundle-registry: v2.5.0-63 - IndexImage: registry-proxy.engineering.redhat.com/rh-osbs/iib:3555
Now we are requiring node-maintenance-operator as an external operator, did you also configured a catalog source for its index image?
I deployed only the CatalogSource for CNV 2.5, not for NMO. NMO is missing from rhos-osbs/iib, which Index Image should be used for NMO?
New try with a CatalogSource added for NMO. - OCP: v4.6.0-fc.1 - CNV-v2.5: registry-proxy.engineering.redhat.com/rh-osbs/iib:4361 (hco-bundle-registry:v2.5.0-84) - NMO-v4.6: registry-proxy.engineering.redhat.com/rh-osbs/iib:4255 > kubectl -n openshift-operator-lifecycle-manager logs deployment/catalog-operator > > E0824 21:13:48.064507 1 queueinformer_operator.go:290] sync "openshift-cnv" failed: found more than one head for channel > I0824 21:13:48.064790 1 event.go:278] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"openshift-cnv", UID:"48083152-a112-428a-9fae-6b9636a65c4a", APIVersion:"v1", ResourceVersion:"182005", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' found more than one head for channel Should I deploy NMO manually before installing CNV?
Can you please share the output of `oc get catalogsources -n openshift-marketplace`?
> oc get catalogsources -n openshift-marketplace > > NAME DISPLAY TYPE PUBLISHER AGE > cnv-25-catalogsource CNV-v2.5 grpc Red Hat 20h > nmo-46-catalogsource NMO-v4.6 grpc Red Hat 20h > redhat-operators-appregistry Red Hat Operators AppRegistry grpc Red Hat 23h - cnv-25-catalogsource uses registry-proxy.engineering.redhat.com/rh-osbs/iib:4361 (hco-bundle-registry:v2.5.0-84) - nmo-46-catalogsource uses registry-proxy.engineering.redhat.com/rh-osbs/iib:4255 - redhat-operators-appregistry uses a catalog image built from redhat-operators AppRegistry (to install LSO)
The conflict is here: redhat-operators-appregistry the head of 2.4 channel on redhat-operators-appregistry (released content) is 2.4.0 while cnv-25-catalogsource and nmo-46-catalogsource are built starting from staging index image where we already have 2.4.1 and so the conflict. This is going to be implicitly solved once we will officially release 2.4.1. I'm going to open a bug on OLM because redhat-operators-appregistry should be definitively disabled as other default sources in order to be able to perform testing on unreleased content.
I am the one who created and enabled redhat-operators-appregistry to be able to install the local-storage-operator. It's not an OLM issue. I will retry after disabling it.
Still the same issue after disabling redhat-operators-appregistry. cnv-25-catalogsource and nmo-46-catalogsource seems to have the same head for all channels. cnv-25-catalogsource: 2.1|kubevirt-hyperconverged|kubevirt-hyperconverged-operator.v2.1.0 2.2|kubevirt-hyperconverged|kubevirt-hyperconverged-operator.v2.2.0 2.3|kubevirt-hyperconverged|kubevirt-hyperconverged-operator.v2.3.0 2.4|kubevirt-hyperconverged|kubevirt-hyperconverged-operator.v2.4.1 2.5|kubevirt-hyperconverged|kubevirt-hyperconverged-operator.v2.5.0 nmo-46-catalogsource: 2.1|kubevirt-hyperconverged|kubevirt-hyperconverged-operator.v2.1.0 2.2|kubevirt-hyperconverged|kubevirt-hyperconverged-operator.v2.2.0 2.3|kubevirt-hyperconverged|kubevirt-hyperconverged-operator.v2.3.0 2.4|kubevirt-hyperconverged|kubevirt-hyperconverged-operator.v2.4.1
take a look at this after 2.4.1 to see if it will resolve itself
Created attachment 1712987 [details] Operators page
Created attachment 1712988 [details] HCO CR
Created attachment 1712989 [details] Pods
After further investigation it came out that the real issue is https://bugzilla.redhat.com/show_bug.cgi?id=1866861 : An index image built with 4.5 opm tool (like the CVP built ones) currently doesn't work on OCP 4.6. My workaround of pruning the two index images worked just because I sued an up-to-date opm tool from 4.6 to do that. So we have to wait for #1866861 to be fixed and in the meanwhile touch and republish the index images we get from CVP with opm from 4.6.
The "found more than one head for channel" issue is fixed in registry-proxy.engineering.redhat.com/rh-osbs/iib:6196. We now need to have the NMO included in the same IIB image as CNV to not have to create a CatalogSource and an ImageContentSourcePolicy (triggering yet another cluster reboot) just for NMO.
The "found more than one head for channel" issue appeared again with registry-proxy.engineering.redhat.com/rh-osbs/iib:6483 (hco-bundle-registry:v2.5.0-139).
Yes, registry-proxy.engineering.redhat.com/rh-osbs/iib:6196 has been "manually" built forcing IIB to use 4.6 binaries, we still have to find the right way to automate it.
deployment works now
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Virtualization 2.5.0 Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:5127