Bug 2255315

Summary: [Backport to 4.14.z] odf-operator should update it's upgradeable conditions based on the dependents
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Nitin Goyal <nigoyal>
Component: odf-operatorAssignee: Nitin Goyal <nigoyal>
Status: CLOSED ERRATA QA Contact: Jilju Joy <jijoy>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.14CC: bkunal, ebenahar, lgangava, muagarwa, nberry, odf-bz-bot
Target Milestone: ---   
Target Release: ODF 4.14.6   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 4.14.6-1 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 2255310
: 2256903 (view as bug list) Environment:
Last Closed: 2024-04-01 09:17:40 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2255310    
Bug Blocks: 2256903, 2256904    

Description Nitin Goyal 2023-12-20 03:48:11 UTC 
+++ This bug was initially created as a clone of Bug #2255310 +++

Description of problem (please be detailed as possible and provide log
snippests):
odf-operator should update it's upgradeable conditions based on the dependents

Version of all relevant components (if applicable):
4.14

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?
yes


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Install odf-operator and observe it's dependents (ocs, mcg and csi-addons) are also installed
2. Update either mcg or csi-addon operator condition (oc get condition) for not upgradeable status, can use a cmd like
> oc -nopenshift-storage patch condition <mcg-name> --type merge -p '{"spec": {"conditions": [{"lastTransitionTime":"2023-12-19T04:06:22Z","message":"MSG","reason":"RSN","status":"False","type":"Upgradeable"}]}}'


Actual results:
- odf-operator condition is not reflecting not upgradeable status

Expected results:
- odf-operator condition should reflect not upgradeable status

Additional info:
- during testing once odf-operator reflects the correct status, remove the condition on it's dependent and odf-operator also should update it's condition
Comment 8 Jilju Joy 2024-03-25 12:07:59 UTC 
Verified in version:

$ oc get csv -n openshift-storage
NAME                                    DISPLAY                       VERSION        REPLACES                                PHASE
mcg-operator.v4.14.6-rhodf              NooBaa Operator               4.14.6-rhodf   mcg-operator.v4.14.5-rhodf              Succeeded
ocs-operator.v4.14.6-rhodf              OpenShift Container Storage   4.14.6-rhodf   ocs-operator.v4.14.5-rhodf              Succeeded
odf-csi-addons-operator.v4.14.6-rhodf   CSI Addons                    4.14.6-rhodf   odf-csi-addons-operator.v4.14.5-rhodf   Succeeded
odf-operator.v4.14.6-rhodf              OpenShift Data Foundation     4.14.6-rhodf   odf-operator.v4.14.5-rhodf              Succeeded

$ oc get csv odf-operator.v4.14.6-rhodf -o yaml | grep full_version
    full_version: 4.14.6-1
    
$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.14.0-0.nightly-2024-03-24-035901   True        False         4h12m   Cluster version is 4.14.0-0.nightly-2024-03-24-035901


Initial values:

$ oc get condition -n openshift-storage
NAME                                    AGE
mcg-operator.v4.14.6-rhodf              4h7m
ocs-operator.v4.14.6-rhodf              4h7m
odf-csi-addons-operator.v4.14.6-rhodf   4h7m
odf-operator.v4.14.6-rhodf              4h7m


$ oc get condition odf-operator.v4.14.6-rhodf -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-03-25T07:40:42Z"
  generation: 8
  labels:
    operators.coreos.com/odf-operator.openshift-storage: ""
  name: odf-operator.v4.14.6-rhodf
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: odf-operator.v4.14.6-rhodf
    uid: 833ebf61-ad52-4a4b-b227-55e7829d8cdc
  resourceVersion: "42633"
  uid: 95afa9aa-ffed-41b1-a08b-7e8f1f3b4e14
spec:
  conditions:
  - lastTransitionTime: "2024-03-25T07:49:07Z"
    message: No dependent reports not upgradeable status
    reason: Dependents
    status: "True"
    type: Upgradeable
  deployments:
  - odf-operator-controller-manager
  - odf-console
  serviceAccounts:
  - odf-operator-controller-manager
  - odf-operator-controller-manager
status:
  conditions:
  - lastTransitionTime: "2024-03-25T07:49:07Z"
    message: No dependent reports not upgradeable status
    observedGeneration: 8
    reason: Dependents
    status: "True"
    type: Upgradeable
    
    
$ oc get condition ocs-operator.v4.14.6-rhodf -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-03-25T07:40:18Z"
  generation: 8
  labels:
    operators.coreos.com/ocs-operator.openshift-storage: ""
  name: ocs-operator.v4.14.6-rhodf
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: ocs-operator.v4.14.6-rhodf
    uid: 45aed859-7002-408e-93c0-5ebd4d7e3b0e
  resourceVersion: "42630"
  uid: 61df9f5a-04cf-4cae-9f45-c345ea76e7bd
spec:
  conditions:
  - lastTransitionTime: "2024-03-25T07:49:07Z"
    message: Reconcile completed successfully
    reason: ReconcileCompleted
    status: "True"
    type: Upgradeable
  deployments:
  - ocs-operator
  - rook-ceph-operator
  - ocs-metrics-exporter
  - ux-backend-server
  serviceAccounts:
  - ocs-metrics-exporter
  - ocs-operator
  - ux-backend-server
  - rook-ceph-mgr
  - rook-ceph-osd
  - rook-ceph-system
  - rook-csi-cephfs-plugin-sa
  - rook-csi-cephfs-provisioner-sa
  - rook-csi-nfs-plugin-sa
  - rook-csi-nfs-provisioner-sa
  - rook-csi-rbd-plugin-sa
  - rook-csi-rbd-provisioner-sa
  - rook-ceph-system
  - ocs-metrics-exporter
  - rook-ceph-cmd-reporter
  - rook-ceph-mgr
  - rook-ceph-osd
  - rook-ceph-purge-osd
  - rook-ceph-rgw
  - rook-ceph-system
  - rook-csi-cephfs-provisioner-sa
  - rook-csi-rbd-plugin-sa
  - rook-csi-rbd-provisioner-sa
status:
  conditions:
  - lastTransitionTime: "2024-03-25T07:49:07Z"
    message: Reconcile completed successfully
    observedGeneration: 8
    reason: ReconcileCompleted
    status: "True"
    type: Upgradeable
    

$ oc get condition odf-csi-addons-operator.v4.14.6-rhodf -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-03-25T07:40:40Z"
  generation: 1
  labels:
    operators.coreos.com/odf-csi-addons-operator.openshift-storage: ""
  name: odf-csi-addons-operator.v4.14.6-rhodf
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: odf-csi-addons-operator.v4.14.6-rhodf
    uid: 0ee6d7e4-7fd3-4764-b6d2-458ecc9a89d4
  resourceVersion: "34129"
  uid: 526a7f94-4a85-458b-99d8-4c7a24702710
spec:
  deployments:
  - csi-addons-controller-manager
  serviceAccounts:
  - csi-addons-controller-manager
  - csi-addons-controller-manager
  
  
$ oc get condition mcg-operator.v4.14.6-rhodf -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-03-25T07:40:36Z"
  generation: 1
  labels:
    operators.coreos.com/mcg-operator.openshift-storage: ""
  name: mcg-operator.v4.14.6-rhodf
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: mcg-operator.v4.14.6-rhodf
    uid: aa7b0d07-4375-4e23-b26a-74107f4f8cdf
  resourceVersion: "33879"
  uid: 84ef5eee-4363-484c-8236-417448d2193d
spec:
  deployments:
  - noobaa-operator
  serviceAccounts:
  - noobaa
  - noobaa
  - noobaa-endpoint
  - noobaa-odf-ui
  
  
Step 1:
Change "Upgradeable" to False in mcg-operator condition.

$ oc -n openshift-storage patch condition mcg-operator.v4.14.6-rhodf --type merge -p '{"spec": {"conditions": [{"lastTransitionTime":"2024-03-25T07:40:36Z","message":"TestMessage","reason":"TestReason","status":"False","type":"Upgradeable"}]}}'
operatorcondition.operators.coreos.com/mcg-operator.v4.14.6-rhodf patched


Step2:
Verify the status of odf-operator condition.

The status of odf-operator condition changed immediately.


$ oc get condition odf-operator.v4.14.6-rhodf -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-03-25T07:40:42Z"
  generation: 9
  labels:
    operators.coreos.com/odf-operator.openshift-storage: ""
  name: odf-operator.v4.14.6-rhodf
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: odf-operator.v4.14.6-rhodf
    uid: 833ebf61-ad52-4a4b-b227-55e7829d8cdc
  resourceVersion: "161467"
  uid: 95afa9aa-ffed-41b1-a08b-7e8f1f3b4e14
spec:
  conditions:
  - lastTransitionTime: "2024-03-25T11:53:36Z"
    message: mcg-operator.v4.14.6-rhodf:TestMessage
    reason: TestReason
    status: "False"
    type: Upgradeable
  deployments:
  - odf-operator-controller-manager
  - odf-console
  serviceAccounts:
  - odf-operator-controller-manager
  - odf-operator-controller-manager
status:
  conditions:
  - lastTransitionTime: "2024-03-25T11:53:36Z"
    message: mcg-operator.v4.14.6-rhodf:TestMessage
    observedGeneration: 9
    reason: TestReason
    status: "False"
    type: Upgradeable
    
    
$ oc get condition mcg-operator.v4.14.6-rhodf -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-03-25T07:40:36Z"
  generation: 2
  labels:
    operators.coreos.com/mcg-operator.openshift-storage: ""
  name: mcg-operator.v4.14.6-rhodf
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: mcg-operator.v4.14.6-rhodf
    uid: aa7b0d07-4375-4e23-b26a-74107f4f8cdf
  resourceVersion: "161465"
  uid: 84ef5eee-4363-484c-8236-417448d2193d
spec:
  conditions:
  - lastTransitionTime: "2024-03-25T07:40:36Z"
    message: TestMessage
    reason: TestReason
    status: "False"
    type: Upgradeable
  deployments:
  - noobaa-operator
  serviceAccounts:
  - noobaa
  - noobaa
  - noobaa-endpoint
  - noobaa-odf-ui
status:
  conditions:
  - lastTransitionTime: "2024-03-25T07:40:36Z"
    message: TestMessage
    observedGeneration: 2
    reason: TestReason
    status: "False"
    type: Upgradeable
    

Step 3:
Change "Upgradeable" to False in odf-csi-addons-operator condition.

$ oc -n openshift-storage patch condition odf-csi-addons-operator.v4.14.6-rhodf --type merge -p '{"spec": {"conditions": [{"lastTransitionTime":"2024-03-25T07:40:40Z","message":"CannotUpgradeAddons","reason":"AddonsNotUpgradable","status":"False","type":"Upgradeable"}]}}'
operatorcondition.operators.coreos.com/odf-csi-addons-operator.v4.14.6-rhodf patched


$ oc get condition odf-csi-addons-operator.v4.14.6-rhodf -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-03-25T07:40:40Z"
  generation: 2
  labels:
    operators.coreos.com/odf-csi-addons-operator.openshift-storage: ""
  name: odf-csi-addons-operator.v4.14.6-rhodf
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: odf-csi-addons-operator.v4.14.6-rhodf
    uid: 0ee6d7e4-7fd3-4764-b6d2-458ecc9a89d4
  resourceVersion: "163335"
  uid: 526a7f94-4a85-458b-99d8-4c7a24702710
spec:
  conditions:
  - lastTransitionTime: "2024-03-25T07:40:40Z"
    message: CannotUpgradeAddons
    reason: AddonsNotUpgradable
    status: "False"
    type: Upgradeable
  deployments:
  - csi-addons-controller-manager
  serviceAccounts:
  - csi-addons-controller-manager
  - csi-addons-controller-manager
status:
  conditions:
  - lastTransitionTime: "2024-03-25T07:40:40Z"
    message: CannotUpgradeAddons
    observedGeneration: 2
    reason: AddonsNotUpgradable
    status: "False"
    type: Upgradeable
    
Step 4:
Check the status of odf-operator condition.
The status is still "Upgradeable" set to "False"


$ oc get condition odf-operator.v4.14.6-rhodf -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-03-25T07:40:42Z"
  generation: 9
  labels:
    operators.coreos.com/odf-operator.openshift-storage: ""
  name: odf-operator.v4.14.6-rhodf
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: odf-operator.v4.14.6-rhodf
    uid: 833ebf61-ad52-4a4b-b227-55e7829d8cdc
  resourceVersion: "161467"
  uid: 95afa9aa-ffed-41b1-a08b-7e8f1f3b4e14
spec:
  conditions:
  - lastTransitionTime: "2024-03-25T11:53:36Z"
    message: mcg-operator.v4.14.6-rhodf:TestMessage
    reason: TestReason
    status: "False"
    type: Upgradeable
  deployments:
  - odf-operator-controller-manager
  - odf-console
  serviceAccounts:
  - odf-operator-controller-manager
  - odf-operator-controller-manager
status:
  conditions:
  - lastTransitionTime: "2024-03-25T11:53:36Z"
    message: mcg-operator.v4.14.6-rhodf:TestMessage
    observedGeneration: 9
    reason: TestReason
    status: "False"
    type: Upgradeable
    
    
Step 5:
Change "Upgradeable" to True in mcg-operator condition.

$ oc -n openshift-storage patch condition mcg-operator.v4.14.6-rhodf --type merge -p '{"spec": {"conditions": [{"lastTransitionTime":"2024-03-25T07:40:36Z","message":"TestMessage","reason":"TestReason","status":"True","type":"Upgradeable"}]}}'
operatorcondition.operators.coreos.com/mcg-operator.v4.14.6-rhodf patched

$ oc get condition mcg-operator.v4.14.6-rhodf -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-03-25T07:40:36Z"
  generation: 3
  labels:
    operators.coreos.com/mcg-operator.openshift-storage: ""
  name: mcg-operator.v4.14.6-rhodf
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: mcg-operator.v4.14.6-rhodf
    uid: aa7b0d07-4375-4e23-b26a-74107f4f8cdf
  resourceVersion: "164462"
  uid: 84ef5eee-4363-484c-8236-417448d2193d
spec:
  conditions:
  - lastTransitionTime: "2024-03-25T07:40:36Z"
    message: TestMessage
    reason: TestReason
    status: "True"
    type: Upgradeable
  deployments:
  - noobaa-operator
  serviceAccounts:
  - noobaa
  - noobaa
  - noobaa-endpoint
  - noobaa-odf-ui
status:
  conditions:
  - lastTransitionTime: "2024-03-25T07:40:36Z"
    message: TestMessage
    observedGeneration: 3
    reason: TestReason
    status: "True"
    type: Upgradeable
    

Step 6:
Check the status of odf-operator condition.
The reason changed in odf-operator.v4.14.6-rhodf condition. Currently it is not upgradable because odf-csi-addons-operator.v4.14.6-rhodf status for "Upgradeable" is still False.


$ oc get condition odf-operator.v4.14.6-rhodf -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-03-25T07:40:42Z"
  generation: 10
  labels:
    operators.coreos.com/odf-operator.openshift-storage: ""
  name: odf-operator.v4.14.6-rhodf
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: odf-operator.v4.14.6-rhodf
    uid: 833ebf61-ad52-4a4b-b227-55e7829d8cdc
  resourceVersion: "164465"
  uid: 95afa9aa-ffed-41b1-a08b-7e8f1f3b4e14
spec:
  conditions:
  - lastTransitionTime: "2024-03-25T11:53:36Z"
    message: odf-csi-addons-operator.v4.14.6-rhodf:CannotUpgradeAddons
    reason: AddonsNotUpgradable
    status: "False"
    type: Upgradeable
  deployments:
  - odf-operator-controller-manager
  - odf-console
  serviceAccounts:
  - odf-operator-controller-manager
  - odf-operator-controller-manager
status:
  conditions:
  - lastTransitionTime: "2024-03-25T11:53:36Z"
    message: odf-csi-addons-operator.v4.14.6-rhodf:CannotUpgradeAddons
    observedGeneration: 10
    reason: AddonsNotUpgradable
    status: "False"
    type: Upgradeable
    
    
Step 7:
Change "Upgradeable" to True in odf-csi-addons-operator.v4.14.6-rhodf condition.

$ oc -n openshift-storage patch condition odf-csi-addons-operator.v4.14.6-rhodf --type merge -p '{"spec": {"conditions": [{"lastTransitionTime":"2024-03-25T07:40:40Z","message":"CannotUpgradeAddons","reason":"AddonsNotUpgradable","status":"True","type":"Upgradeable"}]}}'
operatorcondition.operators.coreos.com/odf-csi-addons-operator.v4.14.6-rhodf patched


$ oc get condition odf-csi-addons-operator.v4.14.6-rhodf -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-03-25T07:40:40Z"
  generation: 3
  labels:
    operators.coreos.com/odf-csi-addons-operator.openshift-storage: ""
  name: odf-csi-addons-operator.v4.14.6-rhodf
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: odf-csi-addons-operator.v4.14.6-rhodf
    uid: 0ee6d7e4-7fd3-4764-b6d2-458ecc9a89d4
  resourceVersion: "166173"
  uid: 526a7f94-4a85-458b-99d8-4c7a24702710
spec:
  conditions:
  - lastTransitionTime: "2024-03-25T07:40:40Z"
    message: CannotUpgradeAddons
    reason: AddonsNotUpgradable
    status: "True"
    type: Upgradeable
  deployments:
  - csi-addons-controller-manager
  serviceAccounts:
  - csi-addons-controller-manager
  - csi-addons-controller-manager
status:
  conditions:
  - lastTransitionTime: "2024-03-25T07:40:40Z"
    message: CannotUpgradeAddons
    observedGeneration: 3
    reason: AddonsNotUpgradable
    status: "True"
    type: Upgradeable
    
    
Step 8:
Verify the status of odf-operator condition.
The status of odf-operator condition changed immediately as expected. "Upgradeable" status is "True".


$ oc get condition odf-operator.v4.14.6-rhodf -o yaml
apiVersion: operators.coreos.com/v2
kind: OperatorCondition
metadata:
  creationTimestamp: "2024-03-25T07:40:42Z"
  generation: 11
  labels:
    operators.coreos.com/odf-operator.openshift-storage: ""
  name: odf-operator.v4.14.6-rhodf
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: true
    kind: ClusterServiceVersion
    name: odf-operator.v4.14.6-rhodf
    uid: 833ebf61-ad52-4a4b-b227-55e7829d8cdc
  resourceVersion: "166176"
  uid: 95afa9aa-ffed-41b1-a08b-7e8f1f3b4e14
spec:
  conditions:
  - lastTransitionTime: "2024-03-25T12:03:22Z"
    message: No dependent reports not upgradeable status
    reason: Dependents
    status: "True"
    type: Upgradeable
  deployments:
  - odf-operator-controller-manager
  - odf-console
  serviceAccounts:
  - odf-operator-controller-manager
  - odf-operator-controller-manager
status:
  conditions:
  - lastTransitionTime: "2024-03-25T12:03:22Z"
    message: No dependent reports not upgradeable status
    observedGeneration: 11
    reason: Dependents
    status: "True"
    type: Upgradeable
Comment 12 errata-xmlrpc 2024-04-01 09:17:40 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Data Foundation 4.14.6 Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:1579