Created attachment 1802328 [details] catalog-operator pod log Description of problem: Multiple channel heads are reported when deploying the amq_streams operator. That is not 100% reproducable. I had deployed a customized catalogsource on this cluster which include same packagemanifests. Not sure if that mess up the OLM. Jiazha: I checked their CSV files, looks good, details: http://pastebin.test.redhat.com/979956 Version-Release number of selected component (if applicable): 4.7.20 How reproducible: Sometime Steps to Reproduce: 1. Create customized catalogsource You can make your index image by command 'opm index prune -p amq-streams -f registry-proxy.engineering.redhat.com/rh-osbs/iib-pub-pending:v4.6 -t quay.io/test/test-index:latest-c podman' 2. Deploy amq-stream operator. $git clone git.redhat.com:aosqe/aosqe-tools.git $oc process -f aosqe-tools/logging/log_template/kafka/amqstreams/01_namespace_amqstreams_template.yaml | oc create -f - $oc project amq-aosqe $oc process -f aosqe-tools/logging/log_template/kafka/amqstreams/02_og_amqstreams_template.yaml|oc create -f - oc create -f aosqe-tools/logging/log_template/kafka/amqstreams/03_sub_amq_streams.yaml 3. Check the catalog-operator logs Actual results: oc logs catalog-operator-776ff5bcb8-cs9b9 on: 'ResolutionFailed' found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles time="2021-07-16T04:10:17Z" level=info msg="catalog update required at 2021-07-16 04:10:17.692278974 +0000 UTC m=+10345.049091771" CatalogSource=community-operators time="2021-07-16T04:10:33Z" level=info msg="catalog polling result: no update" CatalogSource=community-operators time="2021-07-16T04:10:48Z" level=info msg="Adding related objects for operator-lifecycle-manager-catalog" time="2021-07-16T04:11:41Z" level=info msg="catalog update required at 2021-07-16 04:11:41.920504045 +0000 UTC m=+10429.277316845" CatalogSource=redhat-operators time="2021-07-16T04:11:58Z" level=info msg="catalog polling result: no update" CatalogSource=redhat-operators time="2021-07-16T04:12:50Z" level=info msg="catalog update required at 2021-07-16 04:12:50.437067181 +0000 UTC m=+10497.793879979" CatalogSource=redhat-marketplace time="2021-07-16T04:13:08Z" level=info msg="catalog polling result: no update" CatalogSource=redhat-marketplace E0716 04:13:36.828509 1 queueinformer_operator.go:290] sync {"update" "x923s"} failed: found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles I0716 04:13:36.828619 1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"x923s", UID:"7e57763a-a7a3-4119-81b4-20fab9ec6d9b", APIVersion:"v1", ResourceVersion:"115395", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles E0716 04:13:37.233898 1 queueinformer_operator.go:290] sync {"update" "x923s"} failed: found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles I0716 04:13:37.233991 1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"x923s", UID:"7e57763a-a7a3-4119-81b4-20fab9ec6d9b", APIVersion:"v1", ResourceVersion:"115395", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles E0716 04:13:37.629346 1 queueinformer_operator.go:290] sync {"update" "x923s"} failed: found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles I0716 04:13:37.629598 1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"x923s", UID:"7e57763a-a7a3-4119-81b4-20fab9ec6d9b", APIVersion:"v1", ResourceVersion:"115395", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles E0716 04:13:38.029789 1 queueinformer_operator.go:290] sync {"update" "x923s"} failed: found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles I0716 04:13:38.029903 1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"x923s", UID:"7e57763a-a7a3-4119-81b4-20fab9ec6d9b", APIVersion:"v1", ResourceVersion:"115395", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles E0716 04:13:38.428829 1 queueinformer_operator.go:290] sync {"update" "x923s"} failed: found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles I0716 04:13:38.428931 1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"x923s", UID:"7e57763a-a7a3-4119-81b4-20fab9ec6d9b", APIVersion:"v1", ResourceVersion:"115395", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles E0716 04:13:38.830818 1 queueinformer_operator.go:290] sync {"update" "x923s"} failed: found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles I0716 04:13:38.830939 1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"x923s", UID:"7e57763a-a7a3-4119-81b4-20fab9ec6d9b", APIVersion:"v1", ResourceVersion:"115395", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles E0716 04:13:39.231109 1 queueinformer_operator.go:290] sync {"update" "x923s"} failed: found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles I0716 04:13:39.231273 1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"x923s", UID:"7e57763a-a7a3-4119-81b4-20fab9ec6d9b", APIVersion:"v1", ResourceVersion:"115395", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles E0716 04:13:39.628397 1 queueinformer_operator.go:290] sync {"update" "x923s"} failed: found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles I0716 04:13:39.628509 1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"x923s", UID:"7e57763a-a7a3-4119-81b4-20fab9ec6d9b", APIVersion:"v1", ResourceVersion:"115395", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles I0716 04:13:40.279349 1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"x923s", UID:"7e57763a-a7a3-4119-81b4-20fab9ec6d9b", APIVersion:"v1", ResourceVersion:"115395", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles time="2021-07-16T04:15:53Z" level=info msg="Adding related objects for operator-lifecycle-manager-catalog" time="2021-07-16T04:17:25Z" level=info msg="catalog update required at 2021-07-16 04:17:25.012306874 +0000 UTC m=+10772.369119663" CatalogSource=certified-operators time="2021-07-16T04:17:36Z" level=info msg="catalog polling result: no update" CatalogSource=certified-operators Expected results: amq-stream operators can be deployed Additional info:
>. Jiazha: I checked their CSV files, looks good, details: http://pastebin.test.redhat.com/979956 Please ignore this analysis since I checked the wrong index image, I should check "registry.redhat.io/redhat/redhat-operator-index:v4.7", not the "quay.io/openshift-qe-optional-operators/ocp4-index:cpaas-logging-371" since this amq-streams subscribe to the "redhat-operators" CatalogSource. Below are the retest steps: [cloud-user@preserve-olm-env jian]$ oc get sub -n amq-aosqe amq-streams -o yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: creationTimestamp: "2021-07-16T05:14:12Z" generation: 1 labels: operators.coreos.com/amq-streams.amq-aosqe: "" name: amq-streams namespace: amq-aosqe resourceVersion: "150082" selfLink: /apis/operators.coreos.com/v1alpha1/namespaces/amq-aosqe/subscriptions/amq-streams uid: 0122c6be-637b-48b5-b494-3a6626239e4b spec: channel: stable installPlanApproval: Automatic name: amq-streams source: redhat-operators sourceNamespace: openshift-marketplace ... [cloud-user@preserve-olm-env jian]$ podman create registry.redhat.io/redhat/redhat-operator-index:v4.7 d55a7350a8a59884786baf24695b74b60dec0962ab54388eb9b49928d83373a1 [cloud-user@preserve-olm-env jian]$ podman d55a7350a8a59884786baf24695b74b60dec0962ab54388eb9b49928d83373a1:/database/index.db rh-47.db Error: unrecognized command `podman d55a7350a8a59884786baf24695b74b60dec0962ab54388eb9b49928d83373a1:/database/index.db` Try 'podman --help' for more information. [cloud-user@preserve-olm-env jian]$ podman cp d55a7350a8a59884786baf24695b74b60dec0962ab54388eb9b49928d83373a1:/database/index.db rh-47.db [cloud-user@preserve-olm-env jian]$ sqlite3 rh-47.db SQLite version 3.26.0 2018-12-01 12:34:55 Enter ".help" for usage hints. sqlite> .header on sqlite> .mode column sqlite> .table ... -------------------------------------------------------------------------------------------------------------------- registry.redhat.io/amq7/amq-streams-rhel7-operator-metadata@sha256:4ae366be201075ebd86d5bf352d5c77cfa13b8faf5eee5093bfd480ca3a392a2 sqlite> select bundlepath from operatorbundle where name = "amqstreams.v1.7.2"; bundlepath -------------------------------------------------------------------------------------------------------------------- registry.redhat.io/amq7/amq-streams-rhel7-operator-metadata@sha256:4ae366be201075ebd86d5bf352d5c77cfa13b8faf5eee5093bfd480ca3a392a2 sqlite> sqlite> select bundlepath from operatorbundle where name = "amqstreams.v1.6.3"; bundlepath -------------------------------------------------------------------------------------------------------------------- registry.redhat.io/amq7/amqstreams-rhel7-operator-metadata@sha256:cc19289ff777153ca9ab3cebc45e4c98e9c083437d5e2b6407244aead6e89312 sqlite> select bundlepath from operatorbundle where name = "amqstreams.v1.6.2"; bundlepath -------------------------------------------------------------------------------------------------------------------- registry.redhat.io/amq7/amqstreams-rhel7-operator-metadata@sha256:0cd1c66ab08812de30b2baecfa4269843cbabbfb1899730bd4e16739df0aba4f sqlite> select bundlepath from operatorbundle where name = "amqstreams.v1.7.1"; bundlepath -------------------------------------------------------------------------------------------------------------------- registry.redhat.io/amq7/amqstreams-rhel7-operator-metadata@sha256:ec906a4abe8e72844e0fd0ebf0abdc312c0b599e22dc3b7c80b9543cdc76b622 sqlite> select bundlepath from operatorbundle where name = "amqstreams.v1.7.0"; bundlepath -------------------------------------------------------------------------------------------------------------------- registry.redhat.io/amq7/amqstreams-rhel7-operator-metadata@sha256:6ef3bf83ee075cff73ecc2df897d747b6d589df423dcb3ac1f493d1535e627eb sqlite> select bundlepath from operatorbundle where name = "amqstreams.v1.6.1"; bundlepath -------------------------------------------------------------------------------------------------------------------- registry.redhat.io/amq7/amqstreams-rhel7-operator-metadata@sha256:ecbf8a70bbb6e16c12800446971a824de1d5963a920e7179e88605681b556c76 sqlite> .quit [cloud-user@preserve-olm-env cpaas]$ podman create --name rhv1.6.2 registry.redhat.io/amq7/amqstreams-rhel7-operator-metadata@sha256:cc19289ff777153ca9ab3cebc45e4c98e9c083437d5e2b6407244aead6e89312 /bin/bash 844c41930b84458269b2770215c24d94a757151ef98624ff21f6f918b757a874 [cloud-user@preserve-olm-env cpaas]$ podman cp rhv1.6.2:/manifests/ rhv1.6.2 [cloud-user@preserve-olm-env cpaas]$ tree rhv1.6.2 rhv1.6.2 |-- amq-streams-clusteroperator.configmap.yaml |-- amq-streams-entityoperator.clusterrole.yaml |-- amq-streams-kafkabridges.crd.yaml |-- amq-streams-kafkabroker.clusterrole.yaml |-- amq-streams-kafkaclient.clusterrole.yaml |-- amq-streams-kafkaconnectors.crd.yaml |-- amq-streams-kafkaconnects2is.crd.yaml |-- amq-streams-kafkaconnects.crd.yaml |-- amq-streams-kafkamirrormakers2.crd.yaml |-- amq-streams-kafkamirrormakers.crd.yaml |-- amq-streams-kafkarebalances.crd.yaml |-- amq-streams-kafkas.crd.yaml |-- amq-streams-kafkatopics.crd.yaml |-- amq-streams-kafkausers.crd.yaml |-- amq-streams-topicoperator.clusterrole.yaml `-- amq-streams.v1.6.3.clusterserviceversion.yaml 0 directories, 16 files [cloud-user@preserve-olm-env cpaas]$ cat rhv1.6.2/amq-streams.v1.6.3.clusterserviceversion.yaml |grep replaces replaces: amqstreams.v1.6.2 Check their "replaces" fileds, espcially, v1.6.3, looks good. Not sure why get this error: E0716 04:13:36.828509 1 queueinformer_operator.go:290] sync {"update" "x923s"} failed: found multiple channel heads: [amqstreams.v1.6.2 amqstreams.v1.7.2], please check the `replaces`/`skipRange` fields of the operator bundles. related code: https://github.com/operator-framework/operator-lifecycle-manager/blob/release-4.7/pkg/controller/registry/resolver/resolver.go#L633-L658 [cloud-user@preserve-olm-env cpaas]$ podman cp rhv1.7.1:/manifests/ rhv1.7.1 [cloud-user@preserve-olm-env cpaas]$ cat rhv1.7.1/bundle.clusterserviceversion.yaml |grep replaces replaces: amqstreams.v1.7.0 [cloud-user@preserve-olm-env cpaas]$ cat rhv1.7.2/bundle.clusterserviceversion.yaml |grep replaces replaces: amqstreams.v1.7.1 [cloud-user@preserve-olm-env cpaas]$ cat rhv1.7.0/bundle.clusterserviceversion.yaml |grep replaces replaces: amqstreams.v1.6.3 [cloud-user@preserve-olm-env cpaas]$ cat rhv1.6.3/amq-streams.v1.6.3.clusterserviceversion.yaml |grep replaces replaces: amqstreams.v1.6.2 [cloud-user@preserve-olm-env cpaas]$ cat rhv1.6.2/amq-streams.v1.6.2.clusterserviceversion.yaml |grep replaces replaces: amqstreams.v1.6.1 [cloud-user@preserve-olm-env cpaas]$ cat rhv1.6.2/amq-streams.v1.6.2.clusterserviceversion.yaml |grep "amqstreams.v1.6.2" name: amqstreams.v1.6.2
*** This bug has been marked as a duplicate of bug 1981832 ***