Bug 2066997

Summary: [4.9.z clone] [External Mode] upgrade fails with connecting to external Ceph cluster
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Mudit Agarwal <muagarwa>
Component: ocs-operatorAssignee: Jose A. Rivera <jrivera>
Status: CLOSED ERRATA QA Contact: Vijay Avuthu <vavuthu>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.9CC: ebenahar, jrivera, kramdoss, madam, mmuench, muagarwa, ocs-bugs, odf-bz-bot, owasserm, sostapov, uchapaga, vavuthu
Target Milestone: ---Keywords: Automation, Regression
Target Release: ODF 4.9.5   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 4.9.5-4 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 2064107 Environment:
Last Closed: 2022-04-05 18:59:28 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2064107    
Bug Blocks:    

Comment 5 Vijay Avuthu 2022-03-25 10:52:50 UTC
Update:
========

tested below scenarios

1. upgrade from 4.9.5-4 to ocs-registry:4.10.0-210

Result: PASS
 	
 	https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster-prod/3629/console

2. upgrade from 4.8 live to 4.9.5-4

Result: PASS

	https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster-prod/3631/console

3. upgrade from 4.9 live to 4.9.5-4

Result: FAIL

    https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster-prod/3630/console

    must gather: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/vavuthuzup-z49fix/vavuthuzup-z49fix_20220325T075031/logs/failed_testcase_ocs_logs_1648197136/test_upgrade_ocs_logs/

$ oc get csv
NAME                  DISPLAY                       VERSION   REPLACES              PHASE
mcg-operator.v4.9.5   NooBaa Operator               4.9.5     mcg-operator.v4.9.4   Succeeded
ocs-operator.v4.9.4   OpenShift Container Storage   4.9.4     ocs-operator.v4.9.3   Replacing
ocs-operator.v4.9.5   OpenShift Container Storage   4.9.5     ocs-operator.v4.9.4   Pending
odf-operator.v4.9.5   OpenShift Data Foundation     4.9.5     odf-operator.v4.9.4   Succeeded

$ oc describe csv ocs-operator.v4.9.5
Name:         ocs-operator.v4.9.5
Namespace:    openshift-storage
Labels:       full_version=4.9.5-5

Status:
  Cleanup:
  Conditions:
    Last Transition Time:  2022-03-25T08:34:42Z
    Last Update Time:      2022-03-25T08:34:42Z
    Message:               requirements not yet checked
    Phase:                 Pending
    Reason:                RequirementsUnknown
    Last Transition Time:  2022-03-25T08:34:42Z
    Last Update Time:      2022-03-25T08:34:42Z
    Message:               operator is not upgradeable: The operator is not upgradeable: StorageCluster is not ready.
    Phase:                 Pending
    Reason:                OperatorConditionNotUpgradeable
  Last Transition Time:    2022-03-25T08:34:42Z
  Last Update Time:        2022-03-25T08:34:42Z
  Message:                 operator is not upgradeable: The operator is not upgradeable: StorageCluster is not ready.
  Phase:                   Pending
  Reason:                  OperatorConditionNotUpgradeable
Events:
  Type    Reason                           Age   From                        Message
  ----    ------                           ----  ----                        -------
  Normal  RequirementsUnknown              34m   operator-lifecycle-manager  requirements not yet checked
  Normal  OperatorConditionNotUpgradeable  34m   operator-lifecycle-manager  operator is not upgradeable: The operator is not upgradeable: StorageCluster is not ready.


$ oc describe storagecluster ocs-external-storagecluster
Name:         ocs-external-storagecluster
Namespace:    openshift-storage
Labels:       <none>
Annotations:  storagesystem.odf.openshift.io/watched-by: ocs-external-storagecluster-storagesystem
              uninstall.ocs.openshift.io/cleanup-policy: delete
              uninstall.ocs.openshift.io/mode: graceful

Last Heartbeat Time:   2022-03-25T09:09:43Z
    Last Transition Time:  2022-03-25T08:23:56Z
    Message:               Reconcile completed successfully
    Reason:                ReconcileCompleted
    Status:                True
    Type:                  Upgradeable
    Last Heartbeat Time:   2022-03-25T08:21:56Z
    Last Transition Time:  2022-03-25T08:21:51Z
    Message:               External CephCluster is trying to connect: Attempting to connect to an external Ceph cluster
    Reason:                ExternalClusterStateConnecting
    Status:                True
    Type:                  ExternalClusterConnecting
    Last Heartbeat Time:   2022-03-25T08:21:56Z
    Last Transition Time:  2022-03-25T08:21:51Z
    Message:               External CephCluster is trying to connect: Attempting to connect to an external Ceph cluster
    Reason:                ExternalClusterStateConnecting
    Status:                False
    Type:                  ExternalClusterConnected
  External Secret Hash:    247cd30c6e20f435bef14e171e828aa7394f45b8d7f4fd22fdf5d9772c2b7b45073b3a709296c8b0ef7aed18b4723139efba3d7ede6bd620966f57773282cd78
  Images:
    Ceph:
      Desired Image:  registry.redhat.io/rhceph/rhceph-5-rhel8@sha256:2296c19fbd3a0be84d6030dff789ce3e79b38cc30c39f45913aec97967b65cce
    Noobaa Core:
      Actual Image:   registry.redhat.io/odf4/mcg-core-rhel8@sha256:62c322de8d9818008b846778b2e2b51dede990c9cbf33cfe9dae23d744ec80d2
      Desired Image:  registry.redhat.io/odf4/mcg-core-rhel8@sha256:62c322de8d9818008b846778b2e2b51dede990c9cbf33cfe9dae23d744ec80d2
    Noobaa DB:
      Actual Image:   registry.redhat.io/rhel8/postgresql-12@sha256:b006af9af125adce28546d0ed9d77421abe54182741f6b5e1d31c88dd763580b
      Desired Image:  registry.redhat.io/rhel8/postgresql-12@sha256:b006af9af125adce28546d0ed9d77421abe54182741f6b5e1d31c88dd763580b
  Phase:              Ready
  Related Objects:
    API Version:       ceph.rook.io/v1
    Kind:              CephCluster
    Name:              ocs-external-storagecluster-cephcluster
    Namespace:         openshift-storage
    Resource Version:  56534
    UID:               d20d1481-8b86-47e7-98b9-15b5493982d0
    API Version:       noobaa.io/v1alpha1
    Kind:              NooBaa
    Name:              noobaa
    Namespace:         openshift-storage
    Resource Version:  56616
    UID:               59b49ac3-4775-4bf0-80b5-35db1a0c9893
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  CreationSucceeded  48m   StorageCluster controller  StorageSystem ocs-external-storagecluster-storagesystem created for the StorageCluster ocs-external-storagecluster.

$ oc get cephcluster
NAME                                      DATADIRHOSTPATH   MONCOUNT   AGE   PHASE       MESSAGE                          HEALTH      EXTERNAL
ocs-external-storagecluster-cephcluster                                49m   Connected   Cluster connected successfully   HEALTH_OK   true

Comment 11 errata-xmlrpc 2022-04-05 18:59:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Data Foundation 4.9.5 Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1237