Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1772743

Summary:	ASB/TSB operators failed to upgrade on 4.2.z for no "olm.skipRange" or "replace" field in CSVs
Product:	OpenShift Container Platform	Reporter:	Cuiping HUO <chuo>
Component:	Service Broker	Assignee:	Fabian von Feilitzsch <fabian>
Status:	CLOSED ERRATA	QA Contact:	Cuiping HUO <chuo>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	4.2.z	CC:	aos-bugs, chezhang, fabian, jesusr, jfan, jiazha
Target Milestone:	---	Keywords:	TestBlocker, UpcomingSprint, Upgrades
Target Release:	4.2.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1772392	Environment:
Last Closed:	2020-07-01 16:08:20 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1772392
Bug Blocks:

Description Cuiping HUO 2019-11-15 03:33:50 UTC

+++ This bug was initially created as a clone of Bug #1772392 +++

Description of problem:
ASB/TSB operators failed to upgrade from 4.2 to 4.3 for no "olm.skipRange" or "replace" field in CSVs.

Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2019-11-13-233341

How reproducible:
Always

Steps to Reproduce:
1. change app registry to redhat-operators-art
2. install tsb operators with manifest from redhat-operators-art, choose channel "4.2", the csv version is "openshifttemplateservicebrokeroperator.4.2.5-201911121709"
3. upgrade cluster from 4.2.5 to 4.3.0-0.nightly-2019-11-13-233341
4. change tsb operator subscription channel to "4.3"

Actual results:
the tsb operator subscription channel changed to "4.3", but the csv is still openshifttemplateservicebrokeroperator.4.2.5-201911121709 


Expected results:
the tsb operator subscription channel changed to "4.3", and the csv should be openshifttemplateservicebrokeroperator.openshiftansibleservicebroker.4.3.0-201911121117

Additional info:
checked csv of tsb, there is no "olm.skipRange" for channel "4.2"and "4.3",but "4.1" do have this field. Check with app registry:redhat-operators, there is also no "olm.skipRange" for channel "4.2" but "4.1" have. Attached a detailed log in the attachment.

$oc get packagemanifest openshifttemplateservicebroker -n openshift-marketplace -o yaml
apiVersion: packages.operators.coreos.com/v1
kind: PackageManifest
metadata:
  creationTimestamp: "2019-11-14T02:16:27Z"
  labels:
    catalog: qe-app-registry
    catalog-namespace: openshift-marketplace
    olm-visibility: hidden
    openshift-marketplace: "true"
    opsrc-datastore: "true"
    opsrc-owner-name: qe-app-registry
    opsrc-owner-namespace: openshift-marketplace
    provider: Red Hat, Inc.
    provider-url: ""
  name: openshifttemplateservicebroker
  namespace: openshift-marketplace
  selfLink: /apis/packages.operators.coreos.com/v1/namespaces/openshift-marketplace/packagemanifests/openshifttemplateservicebroker
spec: {}
status:
  catalogSource: qe-app-registry
  catalogSourceDisplayName: ""
  catalogSourceNamespace: openshift-marketplace
  catalogSourcePublisher: ""
  channels:
  - currentCSV: openshifttemplateservicebrokeroperator.4.2.5-201911121709
    currentCSVDesc:
      annotations:
        alm-examples: '[{"apiVersion":"osb.openshift.io/v1","kind":"TemplateServiceBroker","metadata":{"name":"template-service-broker","namespace":"template-service-broker"},"spec":{}}]'
        capabilities: Seamless Upgrades
        containerImage: image-registry.openshift-image-registry.svc:5000/openshift/ose-template-service-broker@sha256:4b33168a9d5963c83f678f1efc0291560ff2689ee3b3f4a1b72d10d02f89a666
        description: OpenShift Template Service Broker is an implementation of the
          [Open Service Broker API](https://github.com/openservicebrokerapi/servicebroker)
      apiservicedefinitions: {}
      customresourcedefinitions:
        owned:
        - description: An Open Service Broker supporting management of OpenShift templates.
          displayName: Template Service Broker
          kind: TemplateServiceBroker
          name: templateservicebrokers.osb.openshift.io
          version: v1
      description: |
        The OpenShift Template Service Broker implements the [Open Service Broker
        API](https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md)
        endpoints:
        - *Catalog*: returns a list of available templates as OSB API
          *Service* objects (the templates are read from one or more
          namespaces configured in the master config).
        - *Provision*: provision a given template (referred by its UID) into a
          namespace.  Under the covers, this creates a non-namespaced
          **BrokerTemplateInstance** object for the template service broker to
          store state associated with the the instantiation, as well as the
          **Secret** and **TemplateInstance** objects which are picked up by
          the **TemplateInstance** controller.  *Provision* is an asynchronous
          operation: it may return before provisioning is completed, and the
          provision status can (must) be recovered via the *Last Operation*
          endpoint (see below).
        - *Bind*: for a given template, return "credentials" exposed in any
          created ConfigMap, Secret, Service or Route object (see
          ExposeAnnotationPrefix and Base64ExposeAnnotationPrefix
          documentation).  The *Bind* call records the fact that it took
          place in the appropriate **BrokerTemplateInstance** object.
        - *Unbind*: this simply removes the metadata previously placed in the
          **BrokerTemplateInstance** object by a *Bind* call.
        - *Deprovision*: removes the objects created by the *Provision* call.
          The garbage collector removes all additional objects created by the
          **TemplateInstance** controller, hopefully transitively, as
          documented above.
        - *Last Operation*: returns the status of the previously run
          asynchronous operation.  In the template service broker, *Provision*
          is the only asynchronous operation.
      displayName: OpenShift Template Service Broker Operator
      installModes:
      - supported: true
        type: OwnNamespace
      - supported: true
        type: SingleNamespace
      - supported: false
        type: MultiNamespace
      - supported: false
        type: AllNamespaces
      provider:
        name: Red Hat, Inc.
      version: 4.2.5-201911121709
    name: "4.2"
  - currentCSV: openshifttemplateservicebrokeroperator.4.3.0-201911081022
    currentCSVDesc:
      annotations:
        alm-examples: '[{"apiVersion":"osb.openshift.io/v1","kind":"TemplateServiceBroker","metadata":{"name":"template-service-broker","namespace":"template-service-broker"},"spec":{}}]'
        capabilities: Seamless Upgrades
        containerImage: image-registry.openshift-image-registry.svc:5000/openshift/ose-template-service-broker@sha256:e79b20346de0faa6e807725ed2989de9aa3d2401a97b77c256bb8359ece316d7
        description: OpenShift Template Service Broker is an implementation of the
          [Open Service Broker API](https://github.com/openservicebrokerapi/servicebroker)
      apiservicedefinitions: {}
      customresourcedefinitions:
        owned:
        - description: An Open Service Broker supporting management of OpenShift templates.
          displayName: Template Service Broker
          kind: TemplateServiceBroker
          name: templateservicebrokers.osb.openshift.io
          version: v1
      description: |
        The OpenShift Template Service Broker implements the [Open Service Broker
        API](https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md)
        endpoints:
        - *Catalog*: returns a list of available templates as OSB API
          *Service* objects (the templates are read from one or more
          namespaces configured in the master config).
        - *Provision*: provision a given template (referred by its UID) into a
          namespace.  Under the covers, this creates a non-namespaced
          **BrokerTemplateInstance** object for the template service broker to
          store state associated with the the instantiation, as well as the
          **Secret** and **TemplateInstance** objects which are picked up by
          the **TemplateInstance** controller.  *Provision* is an asynchronous
          operation: it may return before provisioning is completed, and the
          provision status can (must) be recovered via the *Last Operation*
          endpoint (see below).
        - *Bind*: for a given template, return "credentials" exposed in any
          created ConfigMap, Secret, Service or Route object (see
          ExposeAnnotationPrefix and Base64ExposeAnnotationPrefix
          documentation).  The *Bind* call records the fact that it took
          place in the appropriate **BrokerTemplateInstance** object.
        - *Unbind*: this simply removes the metadata previously placed in the
          **BrokerTemplateInstance** object by a *Bind* call.
        - *Deprovision*: removes the objects created by the *Provision* call.
          The garbage collector removes all additional objects created by the
          **TemplateInstance** controller, hopefully transitively, as
          documented above.
        - *Last Operation*: returns the status of the previously run
          asynchronous operation.  In the template service broker, *Provision*
          is the only asynchronous operation.
      displayName: OpenShift Template Service Broker Operator
      installModes:
      - supported: true
        type: OwnNamespace
      - supported: true
        type: SingleNamespace
      - supported: false
        type: MultiNamespace
      - supported: false
        type: AllNamespaces
      provider:
        name: Red Hat, Inc.
      version: 4.3.0-201911081022
    name: "4.3"
  - currentCSV: openshifttemplateservicebrokeroperator.4.1.24-201911120311
    currentCSVDesc:
      annotations:
        alm-examples: '[{"apiVersion":"osb.openshift.io/v1","kind":"TemplateServiceBroker","metadata":{"name":"template-service-broker","namespace":"template-service-broker"},"spec":{}}]'
        capabilities: Seamless Upgrades
        containerImage: image-registry.openshift-image-registry.svc:5000/openshift/ose-template-service-broker@sha256:b76eaf9a9a73ed95f3c596e1f3723c9ebd4fa9648a4958099c39ac2008058673
        description: OpenShift Template Service Broker is an implementation of the
          [Open Service Broker API](https://github.com/openservicebrokerapi/servicebroker)
        olm.skipRange: '>=4.1.0 <4.1.24-201911120311'
      apiservicedefinitions: {}

--- Additional comment from Cuiping HUO on 2019-11-15 02:28:37 UTC ---

@fvonfeil,
TAB has same issue, we also need tsb to be fixed.

--- Additional comment from Cuiping HUO on 2019-11-15 02:29:13 UTC ---

(In reply to Cuiping HUO from comment #1)
> @fvonfeil,
> TAB has same issue, we also need tsb to be fixed.

should be TSB.

Comment 1 Cuiping HUO 2019-11-18 10:26:55 UTC

@fabian,
Could you help to also fix https://github.com/openshift/ansible-service-broker/pull/1257 and https://github.com/openshift/template-service-broker-operator/pull/61 on 4.2 branch? For 4.2.z upgrade is also blocked by the same issue.

Comment 2 Jesus M. Rodriguez 2020-05-15 03:33:01 UTC

Bug has been reviewed this sprint.

Comment 5 Cuiping HUO 2020-06-01 09:03:43 UTC

Verification failed.

asb operator version before upgrade 4.2.33-202005300632, after change asb operator subscription channel to "4.3" through webconsole, asb operator version is 4.3.24-202005300952
$ oc get csv
NAME                                                DISPLAY                                     VERSION               REPLACES                                            PHASE
openshiftansibleservicebroker.4.2.33-202005300632   OpenShift Ansible Service Broker Operator   4.2.33-202005300632                                                       Replacing
openshiftansibleservicebroker.4.3.24-202005300952   OpenShift Ansible Service Broker Operator   4.3.24-202005300952   openshiftansibleservicebroker.4.2.33-202005300632   Failed

$ oc get csv openshiftansibleservicebroker.4.2.33-202005300632 -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  annotations:
    alm-examples: '[{"apiVersion":"osb.openshift.io/v1", "kind":"AutomationBroker",
      "metadata":{"name":"ansible-service-broker","namespace":"ansible-service-broker"},
      "spec":{"createBrokerNamespace":"false","waitForBroker":"false", "registries":
      [{"type": "rhcc", "name": "rhcc", "url": "https://registry.redhat.io", "white_list":
      [".*-apb$"], "auth_type": "secret", "auth_name": "asb-registry-auth"}]}}]'
    capabilities: Seamless Upgrades
    containerImage: quay.io/openshift-qe-optional-operators/ose-ansible-service-broker@sha256:dcbd3e959f30c836c2360bbb9e25475613903c693fae16a3f70da198bbfaf121
    description: OpenShift Ansible Service Broker is an implementation of the [Open
      Service Broker API](https://github.com/openservicebrokerapi/servicebroker)
    olm.operatorGroup: openshift-ansible-service-broker-fx8dj
    olm.operatorNamespace: openshift-ansible-service-broker
    olm.skipRange: '>=4.1.0 <4.2.33-202005300632'
    olm.targetNamespaces: openshift-ansible-service-broker
  creationTimestamp: "2020-06-01T08:51:57Z"
  generation: 1
  labels:
    olm.api.2fc6c4e2d95a77f0: provided
    olm.api.880c523f5d0ea900: provided
    olm.api.7044497f8b7e10a9: provided
    olm.api.dba600531489ce6c: provided
  name: openshiftansibleservicebroker.4.2.33-202005300632
  namespace: openshift-ansible-service-broker
  resourceVersion: "57853"
  selfLink: /apis/operators.coreos.com/v1alpha1/namespaces/openshift-ansible-service-broker/clusterserviceversions/openshiftansibleservicebroker.4.2.33-202005300632
  uid: 211f86d3-4808-424a-b570-6bbee7a3c83b
spec:
  apiservicedefinitions: {}

$ oc get csv openshiftansibleservicebroker.4.3.24-202005300952 -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  annotations:
    alm-examples: '[{"apiVersion":"osb.openshift.io/v1", "kind":"AutomationBroker",
      "metadata":{"name":"ansible-service-broker","namespace":"ansible-service-broker"},
      "spec":{"createBrokerNamespace":"false","waitForBroker":"false", "registries":
      [{"type": "rhcc", "name": "rhcc", "url": "https://registry.redhat.io", "white_list":
      [".*-apb$"], "auth_type": "secret", "auth_name": "asb-registry-auth"}]}}]'
    capabilities: Seamless Upgrades
    containerImage: quay.io/openshift-qe-optional-operators/ose-ansible-service-broker@sha256:d9597ed67bec4b3d4b60413fa313698abe280e5f4cc0c7ec290b6bc6d4bcfe3e
    description: OpenShift Ansible Service Broker is an implementation of the [Open
      Service Broker API](https://github.com/openservicebrokerapi/servicebroker)
    olm.operatorGroup: openshift-ansible-service-broker-fx8dj
    olm.operatorNamespace: openshift-ansible-service-broker
    olm.skipRange: '>=4.2.0 <4.3.24-202005300952'
    olm.targetNamespaces: openshift-ansible-service-broker
  creationTimestamp: "2020-06-01T08:53:04Z"
  generation: 1
  labels:
    olm.api.2fc6c4e2d95a77f0: provided
    olm.api.880c523f5d0ea900: provided
    olm.api.7044497f8b7e10a9: provided
    olm.api.dba600531489ce6c: provided
  name: openshiftansibleservicebroker.4.3.24-202005300952
  namespace: openshift-ansible-service-broker
...

status:
  conditions:
  - lastTransitionTime: "2020-06-01T08:58:15Z"
    lastUpdateTime: "2020-06-01T08:58:15Z"
    message: 'install strategy failed: Deployment.apps "openshift-ansible-service-broker-operator"
      is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"name":"openshift-ansible-service-broker-operator"},
      MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable'
    phase: Failed
    reason: InstallComponentFailed
  - lastTransitionTime: "2020-06-01T08:58:17Z"
    lastUpdateTime: "2020-06-01T08:58:17Z"
    message: 'installing: ComponentMissing: missing deployment with name=openshift-ansible-service-broker-operator'
    phase: Pending
    reason: NeedsReinstall
  - lastTransitionTime: "2020-06-01T08:58:17Z"
    lastUpdateTime: "2020-06-01T08:58:17Z"
    message: all requirements found, attempting install
    phase: InstallReady
    reason: AllRequirementsMet
  - lastTransitionTime: "2020-06-01T08:58:19Z"
    lastUpdateTime: "2020-06-01T08:58:19Z"
    message: 'install strategy failed: Deployment.apps "openshift-ansible-service-broker-operator"
      is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"name":"openshift-ansible-service-broker-operator"},
      MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable'
    phase: Failed
    reason: InstallComponentFailed
  - lastTransitionTime: "2020-06-01T08:58:21Z"
    lastUpdateTime: "2020-06-01T08:58:21Z"
    message: 'installing: ComponentMissing: missing deployment with name=openshift-ansible-service-broker-operator'
    phase: Pending
    reason: NeedsReinstall
  - lastTransitionTime: "2020-06-01T08:58:22Z"
    lastUpdateTime: "2020-06-01T08:58:22Z"
    message: all requirements found, attempting install
    phase: InstallReady
    reason: AllRequirementsMet
  - lastTransitionTime: "2020-06-01T08:58:24Z"
    lastUpdateTime: "2020-06-01T08:58:24Z"
    message: 'install strategy failed: Deployment.apps "openshift-ansible-service-broker-operator"
      is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"name":"openshift-ansible-service-broker-operator"},
      MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable'
    phase: Failed
    reason: InstallComponentFailed
  - lastTransitionTime: "2020-06-01T08:58:25Z"
    lastUpdateTime: "2020-06-01T08:58:25Z"
    message: 'installing: ComponentMissing: missing deployment with name=openshift-ansible-service-broker-operator'
    phase: Pending
    reason: NeedsReinstall
  - lastTransitionTime: "2020-06-01T08:58:26Z"
    lastUpdateTime: "2020-06-01T08:58:26Z"
    message: all requirements found, attempting install
    phase: InstallReady
    reason: AllRequirementsMet
  - lastTransitionTime: "2020-06-01T08:58:28Z"
    lastUpdateTime: "2020-06-01T08:58:28Z"
    message: 'install strategy failed: Deployment.apps "openshift-ansible-service-broker-operator"
      is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"name":"openshift-ansible-service-broker-operator"},
      MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable'
    phase: Failed
    reason: InstallComponentFailed
  - lastTransitionTime: "2020-06-01T08:58:30Z"
    lastUpdateTime: "2020-06-01T08:58:30Z"
    message: 'installing: ComponentMissing: missing deployment with name=openshift-ansible-service-broker-operator'
    phase: Pending
    reason: NeedsReinstall
  - lastTransitionTime: "2020-06-01T08:58:31Z"
    lastUpdateTime: "2020-06-01T08:58:31Z"
    message: all requirements found, attempting install
    phase: InstallReady
    reason: AllRequirementsMet
  - lastTransitionTime: "2020-06-01T08:58:32Z"
    lastUpdateTime: "2020-06-01T08:58:32Z"
    message: 'install strategy failed: Deployment.apps "openshift-ansible-service-broker-operator"
      is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"name":"openshift-ansible-service-broker-operator"},
      MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable'
    phase: Failed
    reason: InstallComponentFailed
  - lastTransitionTime: "2020-06-01T08:58:34Z"
    lastUpdateTime: "2020-06-01T08:58:34Z"
    message: 'installing: ComponentMissing: missing deployment with name=openshift-ansible-service-broker-operator'
    phase: Pending
    reason: NeedsReinstall
  - lastTransitionTime: "2020-06-01T08:58:35Z"
    lastUpdateTime: "2020-06-01T08:58:35Z"
    message: all requirements found, attempting install
    phase: InstallReady
    reason: AllRequirementsMet
  - lastTransitionTime: "2020-06-01T08:58:37Z"
    lastUpdateTime: "2020-06-01T08:58:37Z"
    message: 'install strategy failed: Deployment.apps "openshift-ansible-service-broker-operator"
      is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"name":"openshift-ansible-service-broker-operator"},
      MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable'
    phase: Failed
    reason: InstallComponentFailed
  - lastTransitionTime: "2020-06-01T08:58:39Z"
    lastUpdateTime: "2020-06-01T08:58:39Z"
    message: 'installing: ComponentMissing: missing deployment with name=openshift-ansible-service-broker-operator'
    phase: Pending
    reason: NeedsReinstall
  - lastTransitionTime: "2020-06-01T08:58:39Z"
    lastUpdateTime: "2020-06-01T08:58:39Z"
    message: all requirements found, attempting install
    phase: InstallReady
    reason: AllRequirementsMet
  - lastTransitionTime: "2020-06-01T08:58:42Z"
    lastUpdateTime: "2020-06-01T08:58:42Z"
    message: 'install strategy failed: Deployment.apps "openshift-ansible-service-broker-operator"
      is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"name":"openshift-ansible-service-broker-operator"},
      MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable'
    phase: Failed
    reason: InstallComponentFailed
  - lastTransitionTime: "2020-06-01T08:58:44Z"
    lastUpdateTime: "2020-06-01T08:58:44Z"
    message: 'installing: ComponentMissing: missing deployment with name=openshift-ansible-service-broker-operator'
    phase: Pending
    reason: NeedsReinstall
  lastTransitionTime: "2020-06-01T08:58:44Z"
  lastUpdateTime: "2020-06-01T08:58:44Z"
  message: 'installing: ComponentMissing: missing deployment with name=openshift-ansible-service-broker-operator'
  phase: Pending
  reason: NeedsReinstall
  requirementStatus:
  - group: apiextensions.k8s.io
    kind: CustomResourceDefinition
    message: CRD is present and Established condition is true
    name: automationbrokers.osb.openshift.io
    status: Present
    uuid: fbdae634-445b-45d0-8e2a-27dd2dde7acc
    version: v1beta1
  - group: apiextensions.k8s.io
    kind: CustomResourceDefinition
    message: CRD is present and Established condition is true
    name: bundlebindings.automationbroker.io
    status: Present
    uuid: 0149cbc6-50af-4230-931c-675fc20c834f
    version: v1beta1
  - group: apiextensions.k8s.io
    kind: CustomResourceDefinition
    message: CRD is present and Established condition is true
    name: bundleinstances.automationbroker.io
    status: Present
    uuid: c6d85439-e336-4d42-afce-1161b95ca72a
    version: v1beta1
  - group: apiextensions.k8s.io
    kind: CustomResourceDefinition
    message: CRD is present and Established condition is true
    name: bundles.automationbroker.io
    status: Present
    uuid: d47154a8-edde-4426-8899-da5d12610648
    version: v1beta1
  - dependents:
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: namespaced rule:{"verbs":["get"],"apiGroups":[""],"resources":["namespaces"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: namespaced rule:{"verbs":["*"],"apiGroups":[""],"resources":["pods","configmaps","secrets","services","services/finalizers","endpoints"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: namespaced rule:{"verbs":["*"],"apiGroups":["apps"],"resources":["deployments","deployments/finalizers","replicasets"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: namespaced rule:{"verbs":["*"],"apiGroups":["route.openshift.io"],"resources":["routes"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: namespaced rule:{"verbs":["*"],"apiGroups":["apps.openshift.io"],"resources":["deploymentconfigs"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: namespaced rule:{"verbs":["*"],"apiGroups":["monitoring.coreos.com"],"resources":["servicemonitors"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: namespaced rule:{"verbs":["*"],"apiGroups":["monitoring.coreos.com"],"resources":["prometheusrules","servicemonitors"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["*"],"apiGroups":[""],"resources":["serviceaccounts","namespaces","pods"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["create"],"apiGroups":["authentication.k8s.io"],"resources":["tokenreviews"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["create"],"apiGroups":["authorization.k8s.io"],"resources":["subjectaccessreviews"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["create","delete","patch","get","list"],"apiGroups":["rbac.authorization.k8s.io"],"resources":["clusterroles","clusterrolebindings","rolebindings"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["*"],"apiGroups":["servicecatalog.k8s.io"],"resources":["clusterservicebrokers","servicebrokers"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["create","delete","list","get"],"apiGroups":["networking.k8s.io"],"resources":["networkpolicies"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["create"],"apiGroups":["authorization.openshift.io"],"resources":["subjectrulesreview"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["get","update","list"],"apiGroups":["network.openshift.io",""],"resources":["clusternetworks","netnamespaces"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["get","list"],"apiGroups":["image.openshift.io",""],"resources":["images"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["*"],"apiGroups":["osb.openshift.io"],"resources":["*"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["*"],"apiGroups":["automationbroker.io"],"resources":["*"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["get","post","put","patch","delete"],"nonResourceURLs":["/osb","/osb/*"]}
      status: Satisfied
      version: v1beta1
    group: ""
    kind: ServiceAccount
    message: ""
    name: openshift-ansible-service-broker-operator
    status: Present

$ oc logs -f openshift-ansible-service-broker-operator-55fd759f44-v7ml5 
{"level":"info","ts":1591001524.6879263,"logger":"cmd","msg":"Go Version: go1.12.12"}
{"level":"info","ts":1591001524.6883955,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1591001524.6884046,"logger":"cmd","msg":"Version of operator-sdk: v0.10.0+git"}
{"level":"info","ts":1591001524.6884267,"logger":"cmd","msg":"Watching namespace.","Namespace":"openshift-ansible-service-broker"}
{"level":"info","ts":1591001524.8931592,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"osb.openshift.io","Options.Version":"v1","Options.Kind":"AutomationBroker"}
{"level":"info","ts":1591001524.8934033,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"automationbroker-controller","source":"kind source: osb.openshift.io/v1, Kind=AutomationBroker"}
{"level":"info","ts":1591001524.8935764,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1591001525.0922148,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1591001525.1003017,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1591001525.3105023,"logger":"metrics","msg":"Metrics Service object created","Service.Name":"openshift-ansible-service-broker-operator-metrics","Service.Namespace":"openshift-ansible-service-broker"}
{"level":"info","ts":1591001525.311856,"logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"}
{"level":"info","ts":1591001525.4123123,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"automationbroker-controller"}
{"level":"info","ts":1591001525.5124536,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"automationbroker-controller","worker count":1}
{"level":"info","ts":1591001528.8359,"logger":"logging_event_handler","msg":"[playbook task]","name":"ansible-service-broker","namespace":"openshift-ansible-service-broker","gvk":"osb.openshift.io/v1, Kind=AutomationBroker","event_type":"playbook_on_task_start","job":"8674665223082153551","EventData.Name":"ansible-service-broker : Environment Validation"}
{"level":"info","ts":1591001528.9250114,"logger":"logging_event_handler","msg":"[playbook task]","name":"ansible-service-broker","namespace":"openshift-ansible-service-broker","gvk":"osb.openshift.io/v1, Kind=AutomationBroker","event_type":"playbook_on_task_start","job":"8674665223082153551","EventData.Name":"ansible-service-broker : Verify service catalog is installed"}
{"level":"info","ts":1591001529.0049753,"logger":"logging_event_handler","msg":"[playbook task]","name":"ansible-service-broker","namespace":"openshift-ansible-service-broker","gvk":"osb.openshift.io/v1, Kind=AutomationBroker","event_type":"playbook_on_task_start","job":"8674665223082153551","EventData.Name":"ansible-service-broker : Set broker admin cluster rolebinding state=present"}
{"level":"info","ts":1591001530.3362908,"logger":"logging_event_handler","msg":"[playbook task]","name":"ansible-service-broker","namespace":"openshift-ansible-service-broker","gvk":"osb.openshift.io/v1, Kind=AutomationBroker","event_type":"playbook_on_task_start","job":"8674665223082153551","EventData.Name":"ansible-service-broker : fail"}
{"level":"info","ts":1591001530.3685672,"logger":"logging_event_handler","msg":"[playbook task]","name":"ansible-service-broker","namespace":"openshift-ansible-service-broker","gvk":"osb.openshift.io/v1, Kind=AutomationBroker","event_type":"playbook_on_task_start","job":"8674665223082153551","EventData.Name":"ansible-service-broker : fail"}
{"level":"info","ts":1591001530.3983114,"logger":"logging_event_handler","msg":"[playbook task]","name":"ansible-service-broker","namespace":"openshift-ansible-service-broker","gvk":"osb.openshift.io/v1, Kind=AutomationBroker","event_type":"playbook_on_task_start","job":"8674665223082153551","EventData.Name":"ansible-service-broker : Create directory for cert"}
{"level":"info","ts":1591001530.5287209,"logger":"logging_event_handler","msg":"[playbook task]","name":"ansible-service-broker","namespace":"openshift-ansible-service-broker","gvk":"osb.openshift.io/v1, Kind=AutomationBroker","event_type":"playbook_on_task_start","job":"8674665223082153551","EventData.Name":"ansible-service-broker : Create cert"}
{"level":"info","ts":1591001530.5541577,"logger":"logging_event_handler","msg":"[playbook task]","name":"ansible-service-broker","namespace":"openshift-ansible-service-broker","gvk":"osb.openshift.io/v1, Kind=AutomationBroker","event_type":"playbook_on_task_start","job":"8674665223082153551","EventData.Name":"ansible-service-broker : Set tls secret state=present"}
{"level":"info","ts":1591001530.5842347,"logger":"logging_event_handler","msg":"[playbook task]","name":"ansible-service-broker","namespace":"openshift-ansible-service-broker","gvk":"osb.openshift.io/v1, Kind=AutomationBroker","event_type":"playbook_on_task_start","job":"8674665223082153551","EventData.Name":"ansible-service-broker : Remove tls secret"}
{"level":"info","ts":1591001530.617221,"logger":"logging_event_handler","msg":"[playbook task]","name":"ansible-service-broker","namespace":"openshift-ansible-service-broker","gvk":"osb.openshift.io/v1, Kind=AutomationBroker","event_type":"playbook_on_task_start","job":"8674665223082153551","EventData.Name":"ansible-service-broker : Set dashboard redirector state=present"}
{"level":"info","ts":1591001531.5083764,"logger":"proxy","msg":"Watching child resource","kind":"route.openshift.io/v1, Kind=Route","enqueue_kind":"osb.openshift.io/v1, Kind=AutomationBroker"}
{"level":"info","ts":1591001531.5084393,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"automationbroker-controller","source":"kind source: route.openshift.io/v1, Kind=Route"}
{"level":"info","ts":1591001531.9982827,"logger":"logging_event_handler","msg":"[playbook task]","name":"ansible-service-broker","namespace":"openshift-ansible-service-broker","gvk":"osb.openshift.io/v1, Kind=AutomationBroker","event_type":"playbook_on_task_start","job":"8674665223082153551","EventData.Name":"ansible-service-broker : Check if trusted CA configmap exists"}
{"level":"info","ts":1591001533.1325967,"logger":"proxy","msg":"Cache miss: /v1, Kind=ConfigMap, openshift-ansible-service-broker/asb-trusted-ca"}
{"level":"info","ts":1591001533.330897,"logger":"logging_event_handler","msg":"[playbook task]","name":"ansible-service-broker","namespace":"openshift-ansible-service-broker","gvk":"osb.openshift.io/v1, Kind=AutomationBroker","event_type":"playbook_on_task_start","job":"8674665223082153551","EventData.Name":"ansible-service-broker : Create the trusted CA configmap"}
{"level":"info","ts":1591001534.096713,"logger":"proxy","msg":"Cache miss: /v1, Kind=ConfigMap, openshift-ansible-service-broker/asb-trusted-ca"}
{"level":"info","ts":1591001534.1098435,"logger":"proxy","msg":"Injecting owner reference"}
{"level":"info","ts":1591001534.1101308,"logger":"proxy","msg":"Watching child resource","kind":"/v1, Kind=ConfigMap","enqueue_kind":"osb.openshift.io/v1, Kind=AutomationBroker"}
{"level":"info","ts":1591001534.1101623,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"automationbroker-controller","source":"kind source: /v1, Kind=ConfigMap"}
{"level":"info","ts":1591001534.259605,"logger":"logging_event_handler","msg":"[playbook task]","name":"ansible-service-broker","namespace":"openshift-ansible-service-broker","gvk":"osb.openshift.io/v1, Kind=AutomationBroker","event_type":"playbook_on_task_start","job":"8674665223082153551","EventData.Name":"ansible-service-broker : Get trusted CA configmap"}
...


the same failure happens to tsb also.

Comment 8 Cuiping HUO 2020-06-03 08:20:45 UTC

Verified.
asb operator version before upgrade 4.2.33-202005300632, after change asb operator subscription channel to "4.3" through webconsole, asb operator version is 4.3.24-202006011358. Both csvs contain the skipRange 

$ oc get csv -n openshift-ansible-service-broker
NAME                                                DISPLAY                                     VERSION               REPLACES                                            PHASE
openshiftansibleservicebroker.4.2.33-202005300632   OpenShift Ansible Service Broker Operator   4.2.33-202005300632                                                       Replacing
openshiftansibleservicebroker.4.3.24-202006011358   OpenShift Ansible Service Broker Operator   4.3.24-202006011358   openshiftansibleservicebroker.4.2.33-202005300632   Pending

$ oc get csv openshiftansibleservicebroker.4.2.33-202005300632 -o yaml | grep skipRange
    olm.skipRange: '>=4.1.0 <4.2.33-202005300632'
$ oc get csv openshiftansibleservicebroker.4.3.24-202006011358 -o yaml | grep skipRange
    olm.skipRange: '>=4.2.0 <4.3.24-202006011358'

tsb operator version version before upgrade 4.2.33-202005300632, after change tsb operator subscription channel to "4.3" through webconsole, tsb operator version is 4.3.24-202006011358. Both csvs contain the skipRange 

$ oc get csv -n openshift-template-service-broker
NAME                                                         DISPLAY                                      VERSION               REPLACES                                                     PHASE
openshifttemplateservicebrokeroperator.4.2.33-202005300632   OpenShift Template Service Broker Operator   4.2.33-202005300632                                                                Replacing
openshifttemplateservicebrokeroperator.4.3.24-202006011358   OpenShift Template Service Broker Operator   4.3.24-202006011358   openshifttemplateservicebrokeroperator.4.2.33-202005300632   Failed

$ oc get csv openshifttemplateservicebrokeroperator.4.2.33-202005300632 -o yaml | grep skipRange
    olm.skipRange: '>=4.1.0 <4.2.33-202005300632'
$ oc get csv openshifttemplateservicebrokeroperator.4.3.24-202006011358 -o yaml | grep skipRange
    olm.skipRange: '>=4.2.0 <4.3.24-202006011358'

As for the upgrade fail, opening a new bug https://bugzilla.redhat.com/show_bug.cgi?id=1843391 to track.

Comment 10 errata-xmlrpc 2020-07-01 16:08:20 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2589