Bug 1998571
| Summary: | Catalog operator reports multiple heads for several channels | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Ben Luddy <bluddy> |
| Component: | OLM | Assignee: | Ankita Thomas <ankithom> |
| OLM sub component: | OLM | QA Contact: | Jian Zhang <jiazha> |
| Status: | CLOSED INSUFFICIENT_DATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | kbidarka, krizza, rbaumgar, sreber, stirabos, tflannag |
| Version: | 4.8 | Keywords: | Reopened |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-01-17 15:57:45 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Ben Luddy
2021-08-27 15:47:50 UTC
I tried to reproduce this with the 4.8 index, but I haven't been able to. Looking at the redhat-operators index and the community-operators index, it doesn't looks like they have any upgrade graph issues. This could be a transient error with the index at the time, can you provide the subscriptions and catalogs or a must gather? must-gather uploaded. https://drive.google.com/file/d/1vOLG4DpOryWigbhaAcvOjnJYYq3OO6K1/view?usp=sharing ClusterID: d723aa91-ffd9-4578-8ef3-153e92eaee7b ClusterVersion: Stable at "4.8.5" ClusterOperators: clusteroperator/operator-lifecycle-manager is not upgradeable because ClusterServiceVersions blocking cluster upgrade: openshift-operators/openshift-gitops-operator.v1.2.0 is incompatible with OpenShift versions greater than 4.8.0 (In reply to Robert Baumgartner from comment #2) > ClusterID: d723aa91-ffd9-4578-8ef3-153e92eaee7b > ClusterVersion: Stable at "4.8.5" > ClusterOperators: > clusteroperator/operator-lifecycle-manager is not upgradeable because > ClusterServiceVersions blocking cluster upgrade: > openshift-operators/openshift-gitops-operator.v1.2.0 is incompatible with > OpenShift versions greater than 4.8.0 This is expected as OpenShift GitOps 1.2 is not supported on OpenShift Container Platform 4.9. See https://access.redhat.com/solutions/6338121 This is consistently reproducible on a cluster upgraded from 4.7.34 to 4.8.15
And then, when trying to upgrade CNV via CNV_SOURCE=osbs
From fresh cluster today,
]$ oc logs -f catalog-operator-6b54d9586d-zz2pj -n openshift-operator-lifecycle-manager
E1019 05:39:51.700308 1 queueinformer_operator.go:290] sync "openshift-cnv" failed: found multiple channel heads: [kubevirt-hyperconverged-operator.v2.6.4 kubevirt-hyperconverged-operator.v2.6.0 kubevirt-hyperconverged-operator.v4.8.2], please check the `replaces`/`skipRange` fields of the operator bundles
I1019 05:39:51.700475 1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"openshift-cnv", UID:"765c8d6a-db3b-4cb9-8953-88f9113c6913", APIVersion:"v1", ResourceVersion:"131585", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' found multiple channel heads: [kubevirt-hyperconverged-operator.v2.6.4 kubevirt-hyperconverged-operator.v2.6.0 kubevirt-hyperconverged-operator.v4.8.2], please check the `replaces`/`skipRange` fields of the operator bundles
I1019 05:39:52.756466 1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"openshift-cnv", UID:"765c8d6a-db3b-4cb9-8953-88f9113c6913", APIVersion:"v1", ResourceVersion:"131585", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' found multiple channel heads: [kubevirt-hyperconverged-operator.v2.6.4 kubevirt-hyperconverged-operator.v2.6.0 kubevirt-hyperconverged-operator.v4.8.2], please check the `replaces`/`skipRange` fields of the operator bundles
I1019 05:39:53.069015 1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"openshift-cnv", UID:"765c8d6a-db3b-4cb9-8953-88f9113c6913", APIVersion:"v1", ResourceVersion:"131585", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' found multiple channel heads: [kubevirt-hyperconverged-operator.v2.6.4 kubevirt-hyperconverged-operator.v2.6.0 kubevirt-hyperconverged-operator.v4.8.2], please check the `replaces`/`skipRange` fields of the operator bundles
Was expecting a installplan for CNV v4.8.0, but no luck, due to the above issue,
[kbidarka@localhost manifests-iib-1634616320]$ oc get ip -A
NAMESPACE NAME CSV APPROVAL APPROVED
openshift-cnv install-sr8sh kubevirt-hyperconverged-operator.v2.6.7 Manual true
openshift-cnv install-wrknk kubevirt-hyperconverged-operator.v2.6.7 Manual true
openshift-local-storage install-jhg8d local-storage-operator.4.7.0-202110051546 Automatic true
openshift-local-storage install-rc2sk local-storage-operator.4.8.0-202110011559 Automatic true
openshift-storage install-9xdv4 ocs-operator.v4.7.4 Automatic true
openshift-storage install-jmhs4 ocs-operator.v4.7.5 Automatic true
openshift-storage install-k6ntn ocs-operator.v4.8.3 Automatic true
I had to reinstall the setup, as I need to verify a CNV blocker bug. I can reinstall and share the setup, on which this was seen/reproducible. As this is seen with upgrades and upgrades being very time consuming, feel it would be great to have this issue resolved or a at-least we have a workaround. I have hit this issue again today,
The easiest way to reproduce this issue is to install OCP4.7 and also install OCP Virt 2.6.7 from our Jenkins job (which is fully automated. )
I have hit this more consistently on a Bare-Metal setup compared to a PSI Setup, IIRC.
E1113 12:05:17.405753 1 queueinformer_operator.go:290] sync "openshift-cnv" failed: found multiple channel heads: [kubevirt-hyperconverged-operator.v2.6.0 kubevirt-hyperconverged-operator.v2.6.8 kubevirt-hyperconverged-operator.v2.6.4], please check the `replaces`/`skipRange` fields of the operator bundles
I1113 12:05:17.405901 1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"openshift-cnv", UID:"49e63ff0-6e94-45f0-a2ec-af26d71940d7", APIVersion:"v1", ResourceVersion:"47714", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' found multiple channel heads: [kubevirt-hyperconverged-operator.v2.6.0 kubevirt-hyperconverged-operator.v2.6.8 kubevirt-hyperconverged-operator.v2.6.4], please check the `replaces`/`skipRange` fields of the operator bundles
E1113 12:05:17.905954 1 queueinformer_operator.go:290] sync "openshift-cnv" failed: found multiple channel heads: [kubevirt-hyperconverged-operator.v2.6.0 kubevirt-hyperconverged-operator.v2.6.8 kubevirt-hyperconverged-operator.v2.6.4], please check the `replaces`/`skipRange` fields of the operator bundles
I1113 12:05:17.906128 1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"openshift-cnv", UID:"49e63ff0-6e94-45f0-a2ec-af26d71940d7", APIVersion:"v1", ResourceVersion:"47714", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' found multiple channel heads: [kubevirt-hyperconverged-operator.v2.6.0 kubevirt-hyperconverged-operator.v2.6.8 kubevirt-hyperconverged-operator.v2.6.4], please check the `replaces`/`skipRange` fields of the operator bundles
There was an issue with identifying skipped channel head candidates that was fixed in 4.9, backporting that fix to 4.8, 4.7 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days |