Bug 2255310

Summary: odf-operator should update it's upgradeable conditions based on the dependents
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Leela Venkaiah Gangavarapu <lgangava>
Component: odf-operatorAssignee: Nitin Goyal <nigoyal>
Status: CLOSED ERRATA QA Contact: Jilju Joy <jijoy>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.15CC: muagarwa, nberry, nigoyal, odf-bz-bot
Target Milestone: ---Keywords: AutomationBackLog
Target Release: ODF 4.15.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 4.15.0-91 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 2255315 (view as bug list) Environment:
Last Closed: 2024-03-19 15:26:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2255315, 2256903, 2256904    

Description Leela Venkaiah Gangavarapu 2023-12-20 02:57:52 UTC 
Description of problem (please be detailed as possible and provide log
snippests):
odf-operator should update it's upgradeable conditions based on the dependents

Version of all relevant components (if applicable):
4.14

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?
yes


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Install odf-operator and observe it's dependents (ocs, mcg and csi-addons) are also installed
2. Update either mcg or csi-addon operator condition (oc get condition) for not upgradeable status, can use a cmd like
> oc -nopenshift-storage patch condition <mcg-name> --type merge -p '{"spec": {"conditions": [{"lastTransitionTime":"2023-12-19T04:06:22Z","message":"MSG","reason":"RSN","status":"False","type":"Upgradeable"}]}}'


Actual results:
- odf-operator condition is not reflecting not upgradeable status

Expected results:
- odf-operator condition should reflect not upgradeable status

Additional info:
- during testing once odf-operator reflects the correct status, remove the condition on it's dependent and odf-operator also should update it's condition
Comment 6 Jilju Joy 2024-01-17 05:59:55 UTC 
Verified in version:

$ oc get csv -n openshift-storage
NAME                                         DISPLAY                       VERSION             REPLACES   PHASE
mcg-operator.v4.15.0-118.stable              NooBaa Operator               4.15.0-118.stable              Succeeded
ocs-operator.v4.15.0-118.stable              OpenShift Container Storage   4.15.0-118.stable              Succeeded
odf-csi-addons-operator.v4.15.0-118.stable   CSI Addons                    4.15.0-118.stable              Succeeded
odf-operator.v4.15.0-118.stable              OpenShift Data Foundation     4.15.0-118.stable              Succeeded

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.15.0-0.nightly-2024-01-16-113018   True        False         77m     Cluster version is 4.15.0-0.nightly-2024-01-16-113018

Platform is VMware.



Initial:

$ oc get condition -n openshift-storage
NAME                                         AGE
mcg-operator.v4.15.0-118.stable              36m
ocs-operator.v4.15.0-118.stable              36m
odf-csi-addons-operator.v4.15.0-118.stable   36m
odf-operator.v4.15.0-118.stable              36m


$ oc get condition odf-operator.v4.15.0-118.stable -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-01-17T04:47:45Z"
  generation: 6
  labels:
    operators.coreos.com/odf-operator.openshift-storage: ""
  name: odf-operator.v4.15.0-118.stable
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: odf-operator.v4.15.0-118.stable
    uid: 80efa2e9-ae7c-4c55-ae0b-385ae674b353
  resourceVersion: "55492"
  uid: 7726f373-7a59-4bbc-a2d1-45431b75d02c
spec:
  conditions:
  - lastTransitionTime: "2024-01-17T04:57:29Z"
    message: No dependent reports not upgradeable status
    reason: Dependents
    status: "True"
    type: Upgradeable
  deployments:
  - odf-operator-controller-manager
  - odf-console
  serviceAccounts:
  - odf-operator-controller-manager
  - odf-operator-controller-manager
status:
  conditions:
  - lastTransitionTime: "2024-01-17T04:57:29Z"
    message: No dependent reports not upgradeable status
    observedGeneration: 6
    reason: Dependents
    status: "True"
    type: Upgradeable
    
    
$ oc get condition ocs-operator.v4.15.0-118.stable -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-01-17T04:47:50Z"
  generation: 6
  labels:
    operators.coreos.com/ocs-operator.openshift-storage: ""
  name: ocs-operator.v4.15.0-118.stable
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: ocs-operator.v4.15.0-118.stable
    uid: b4f8110b-9de2-4d00-b1eb-aa60548cdfdd
  resourceVersion: "55487"
  uid: b4ab8810-ce35-4bcc-8f6c-0181bea4b9c2
spec:
  conditions:
  - lastTransitionTime: "2024-01-17T04:57:29Z"
    message: Reconcile completed successfully
    reason: ReconcileCompleted
    status: "True"
    type: Upgradeable
  deployments:
  - ocs-operator
  - rook-ceph-operator
  - ux-backend-server
  serviceAccounts:
  - ocs-operator
  - ux-backend-server
  - rook-ceph-mgr
  - rook-ceph-osd
  - rook-ceph-system
  - rook-csi-cephfs-plugin-sa
  - rook-csi-cephfs-provisioner-sa
  - rook-csi-nfs-plugin-sa
  - rook-csi-nfs-provisioner-sa
  - rook-csi-rbd-plugin-sa
  - rook-csi-rbd-provisioner-sa
  - rook-ceph-system
  - rook-ceph-cmd-reporter
  - rook-ceph-mgr
  - rook-ceph-osd
  - rook-ceph-purge-osd
  - rook-ceph-rgw
  - rook-ceph-system
  - rook-csi-cephfs-provisioner-sa
  - rook-csi-rbd-plugin-sa
  - rook-csi-rbd-provisioner-sa
status:
  conditions:
  - lastTransitionTime: "2024-01-17T04:57:29Z"
    message: Reconcile completed successfully
    observedGeneration: 6
    reason: ReconcileCompleted
    status: "True"
    type: Upgradeable


$ oc get condition odf-csi-addons-operator.v4.15.0-118.stable -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-01-17T04:47:34Z"
  generation: 1
  labels:
    operators.coreos.com/odf-csi-addons-operator.openshift-storage: ""
  name: odf-csi-addons-operator.v4.15.0-118.stable
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: odf-csi-addons-operator.v4.15.0-118.stable
    uid: 4cec6b52-e678-4a99-b074-ac289a5e5f10
  resourceVersion: "45445"
  uid: 31d6a231-39de-4aac-8a8f-d9b2e0fabcfd
spec:
  deployments:
  - csi-addons-controller-manager
  serviceAccounts:
  - csi-addons-controller-manager
  - csi-addons-controller-manager
  
  

$ oc get condition mcg-operator.v4.15.0-118.stable -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-01-17T04:47:39Z"
  generation: 1
  labels:
    operators.coreos.com/mcg-operator.openshift-storage: ""
  name: mcg-operator.v4.15.0-118.stable
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: mcg-operator.v4.15.0-118.stable
    uid: 638f9d0d-910f-4011-87a0-633928293d9d
  resourceVersion: "45612"
  uid: 32085afa-ece4-4338-a3d4-e4b9c3594876
spec:
  deployments:
  - noobaa-operator
  serviceAccounts:
  - noobaa
  - noobaa-odf-ui
  - noobaa
  - noobaa-endpoint
  
  

Step 1:
Change "Upgradeable" to False in mcg-operator condition.

$ oc -n openshift-storage patch condition mcg-operator.v4.15.0-118.stable --type merge -p '{"spec": {"conditions": [{"lastTransitionTime":"2024-01-17T04:57:29Z","message":"TestMessage","reason":"TestReason","status":"False","type":"Upgradeable"}]}}'
operatorcondition.operators.coreos.com/mcg-operator.v4.15.0-118.stable patched

Step2:
Verify the status of odf-operator condition.

The status of odf-operator condition changed immediately.

$ oc get condition odf-operator.v4.15.0-118.stable -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-01-17T04:47:45Z"
  generation: 7
  labels:
    operators.coreos.com/odf-operator.openshift-storage: ""
  name: odf-operator.v4.15.0-118.stable
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: odf-operator.v4.15.0-118.stable
    uid: 80efa2e9-ae7c-4c55-ae0b-385ae674b353
  resourceVersion: "74928"
  uid: 7726f373-7a59-4bbc-a2d1-45431b75d02c
spec:
  conditions:
  - lastTransitionTime: "2024-01-17T05:29:20Z"
    message: mcg-operator.v4.15.0-118.stable:TestMessage
    reason: TestReason
    status: "False"
    type: Upgradeable
  deployments:
  - odf-operator-controller-manager
  - odf-console
  serviceAccounts:
  - odf-operator-controller-manager
  - odf-operator-controller-manager
status:
  conditions:
  - lastTransitionTime: "2024-01-17T05:29:20Z"
    message: mcg-operator.v4.15.0-118.stable:TestMessage
    observedGeneration: 7
    reason: TestReason
    status: "False"
    type: Upgradeable
    
    
$ oc get condition mcg-operator.v4.15.0-118.stable -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-01-17T04:47:39Z"
  generation: 2
  labels:
    operators.coreos.com/mcg-operator.openshift-storage: ""
  name: mcg-operator.v4.15.0-118.stable
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: mcg-operator.v4.15.0-118.stable
    uid: 638f9d0d-910f-4011-87a0-633928293d9d
  resourceVersion: "74923"
  uid: 32085afa-ece4-4338-a3d4-e4b9c3594876
spec:
  conditions:
  - lastTransitionTime: "2024-01-17T04:57:29Z"
    message: TestMessage
    reason: TestReason
    status: "False"
    type: Upgradeable
  deployments:
  - noobaa-operator
  serviceAccounts:
  - noobaa
  - noobaa-odf-ui
  - noobaa
  - noobaa-endpoint
status:
  conditions:
  - lastTransitionTime: "2024-01-17T04:57:29Z"
    message: TestMessage
    observedGeneration: 2
    reason: TestReason
    status: "False"
    type: Upgradeable
    
    

Step 3:
Change "Upgradeable" to False in odf-csi-addons-operator condition. 



$ oc -n openshift-storage patch condition odf-csi-addons-operator.v4.15.0-118.stable --type merge -p '{"spec": {"conditions": [{"lastTransitionTime":"2024-01-17T04:57:29Z","message":"CannotUpgradeAddons","reason":"AddonsNotUpgradable","status":"False","type":"Upgradeable"}]}}'
operatorcondition.operators.coreos.com/odf-csi-addons-operator.v4.15.0-118.stable patched


$ oc get condition odf-csi-addons-operator.v4.15.0-118.stable -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-01-17T04:47:34Z"
  generation: 4
  labels:
    operators.coreos.com/odf-csi-addons-operator.openshift-storage: ""
  name: odf-csi-addons-operator.v4.15.0-118.stable
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: odf-csi-addons-operator.v4.15.0-118.stable
    uid: 4cec6b52-e678-4a99-b074-ac289a5e5f10
  resourceVersion: "79006"
  uid: 31d6a231-39de-4aac-8a8f-d9b2e0fabcfd
spec:
  conditions:
  - lastTransitionTime: "2024-01-17T04:57:29Z"
    message: CannotUpgradeAddons
    reason: AddonsNotUpgradable
    status: "False"
    type: Upgradeable
  deployments:
  - csi-addons-controller-manager
  serviceAccounts:
  - csi-addons-controller-manager
  - csi-addons-controller-manager
status:
  conditions:
  - lastTransitionTime: "2024-01-17T04:57:29Z"
    message: CannotUpgradeAddons
    observedGeneration: 4
    reason: AddonsNotUpgradable
    status: "False"
    type: Upgradeable
    
Step 4:
Check the status of odf-operator condition.
The status is still "Upgradeable" set to "False"

$ oc get condition odf-operator.v4.15.0-118.stable -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-01-17T04:47:45Z"
  generation: 7
  labels:
    operators.coreos.com/odf-operator.openshift-storage: ""
  name: odf-operator.v4.15.0-118.stable
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: odf-operator.v4.15.0-118.stable
    uid: 80efa2e9-ae7c-4c55-ae0b-385ae674b353
  resourceVersion: "74928"
  uid: 7726f373-7a59-4bbc-a2d1-45431b75d02c
spec:
  conditions:
  - lastTransitionTime: "2024-01-17T05:29:20Z"
    message: mcg-operator.v4.15.0-118.stable:TestMessage
    reason: TestReason
    status: "False"
    type: Upgradeable
  deployments:
  - odf-operator-controller-manager
  - odf-console
  serviceAccounts:
  - odf-operator-controller-manager
  - odf-operator-controller-manager
status:
  conditions:
  - lastTransitionTime: "2024-01-17T05:29:20Z"
    message: mcg-operator.v4.15.0-118.stable:TestMessage
    observedGeneration: 7
    reason: TestReason
    status: "False"
    type: Upgradeable
    
 
Step 5:
Change "Upgradeable" to True in mcg-operator condition.
 
 
$ oc -n openshift-storage patch condition mcg-operator.v4.15.0-118.stable --type merge -p '{"spec": {"conditions": [{"lastTransitionTime":"2024-01-17T04:57:29Z","message":"McgCanBeUpgraded","reason":"McgIsUpgradable","status":"True","type":"Upgradeable"}]}}'
operatorcondition.operators.coreos.com/mcg-operator.v4.15.0-118.stable patched
 
 
 
 $ oc get condition mcg-operator.v4.15.0-118.stable -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-01-17T04:47:39Z"
  generation: 3
  labels:
    operators.coreos.com/mcg-operator.openshift-storage: ""
  name: mcg-operator.v4.15.0-118.stable
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: mcg-operator.v4.15.0-118.stable
    uid: 638f9d0d-910f-4011-87a0-633928293d9d
  resourceVersion: "81808"
  uid: 32085afa-ece4-4338-a3d4-e4b9c3594876
spec:
  conditions:
  - lastTransitionTime: "2024-01-17T04:57:29Z"
    message: McgCanBeUpgraded
    reason: McgIsUpgradable
    status: "True"
    type: Upgradeable
  deployments:
  - noobaa-operator
  serviceAccounts:
  - noobaa
  - noobaa-odf-ui
  - noobaa
  - noobaa-endpoint
status:
  conditions:
  - lastTransitionTime: "2024-01-17T04:57:29Z"
    message: McgCanBeUpgraded
    observedGeneration: 3
    reason: McgIsUpgradable
    status: "True"
    type: Upgradeable
    
Step 6:
Check the status of odf-operator condition.
The reason changed in odf-operator.v4.15.0-118.stable condition. Currently it is not upgradable because odf-csi-addons-operator.v4.15.0-118.stable status for "Upgradeable" is still False.

$ oc get condition odf-operator.v4.15.0-118.stable -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-01-17T04:47:45Z"
  generation: 8
  labels:
    operators.coreos.com/odf-operator.openshift-storage: ""
  name: odf-operator.v4.15.0-118.stable
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: odf-operator.v4.15.0-118.stable
    uid: 80efa2e9-ae7c-4c55-ae0b-385ae674b353
  resourceVersion: "81810"
  uid: 7726f373-7a59-4bbc-a2d1-45431b75d02c
spec:
  conditions:
  - lastTransitionTime: "2024-01-17T05:29:20Z"
    message: odf-csi-addons-operator.v4.15.0-118.stable:CannotUpgradeAddons
    reason: AddonsNotUpgradable
    status: "False"
    type: Upgradeable
  deployments:
  - odf-operator-controller-manager
  - odf-console
  serviceAccounts:
  - odf-operator-controller-manager
  - odf-operator-controller-manager
status:
  conditions:
  - lastTransitionTime: "2024-01-17T05:29:20Z"
    message: odf-csi-addons-operator.v4.15.0-118.stable:CannotUpgradeAddons
    observedGeneration: 8
    reason: AddonsNotUpgradable
    status: "False"
    type: Upgradeable
    

Step 7:
Change "Upgradeable" to True in odf-csi-addons-operator.v4.15.0-118.stable condition.
  
$ oc -n openshift-storage patch condition odf-csi-addons-operator.v4.15.0-118.stable --type merge -p '{"spec": {"conditions": [{"lastTransitionTime":"2024-01-17T04:57:29Z","message":"AddonsCanBeUpgraded","reason":"AddonIsUpgradable","status":"True","type":"Upgradeable"}]}}'
operatorcondition.operators.coreos.com/odf-csi-addons-operator.v4.15.0-118.stable patched



$ oc get condition odf-csi-addons-operator.v4.15.0-118.stable -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-01-17T04:47:34Z"
  generation: 5
  labels:
    operators.coreos.com/odf-csi-addons-operator.openshift-storage: ""
  name: odf-csi-addons-operator.v4.15.0-118.stable
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: odf-csi-addons-operator.v4.15.0-118.stable
    uid: 4cec6b52-e678-4a99-b074-ac289a5e5f10
  resourceVersion: "85076"
  uid: 31d6a231-39de-4aac-8a8f-d9b2e0fabcfd
spec:
  conditions:
  - lastTransitionTime: "2024-01-17T04:57:29Z"
    message: AddonsCanBeUpgraded
    reason: AddonIsUpgradable
    status: "True"
    type: Upgradeable
  deployments:
  - csi-addons-controller-manager
  serviceAccounts:
  - csi-addons-controller-manager
  - csi-addons-controller-manager
status:
  conditions:
  - lastTransitionTime: "2024-01-17T04:57:29Z"
    message: AddonsCanBeUpgraded
    observedGeneration: 5
    reason: AddonIsUpgradable
    status: "True"
    type: Upgradeable
    
    
Step 8:
Verify the status of odf-operator condition.
The status of odf-operator condition changed immediately as expected. "Upgradeable" status is "True".

$ oc get condition odf-operator.v4.15.0-118.stable -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-01-17T04:47:45Z"
  generation: 9
  labels:
    operators.coreos.com/odf-operator.openshift-storage: ""
  name: odf-operator.v4.15.0-118.stable
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: odf-operator.v4.15.0-118.stable
    uid: 80efa2e9-ae7c-4c55-ae0b-385ae674b353
  resourceVersion: "85078"
  uid: 7726f373-7a59-4bbc-a2d1-45431b75d02c
spec:
  conditions:
  - lastTransitionTime: "2024-01-17T05:46:13Z"
    message: No dependent reports not upgradeable status
    reason: Dependents
    status: "True"
    type: Upgradeable
  deployments:
  - odf-operator-controller-manager
  - odf-console
  serviceAccounts:
  - odf-operator-controller-manager
  - odf-operator-controller-manager
status:
  conditions:
  - lastTransitionTime: "2024-01-17T05:46:13Z"
    message: No dependent reports not upgradeable status
    observedGeneration: 9
    reason: Dependents
    status: "True"
    type: Upgradeable
Comment 7 errata-xmlrpc 2024-03-19 15:26:05 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.15.0 security, enhancement, & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:1383