Bug 2066997 - [4.9.z clone] [External Mode] upgrade fails with connecting to external Ceph cluster
Summary: [4.9.z clone] [External Mode] upgrade fails with connecting to external Ceph ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: ocs-operator
Version: 4.9
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: ODF 4.9.5
Assignee: Jose A. Rivera
QA Contact: Vijay Avuthu
URL:
Whiteboard:
Depends On: 2064107
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-03-23 02:57 UTC by Mudit Agarwal
Modified: 2023-08-09 17:00 UTC (History)
12 users (show)

Fixed In Version: 4.9.5-4
Doc Type: No Doc Update
Doc Text:
Clone Of: 2064107
Environment:
Last Closed: 2022-04-05 18:59:28 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github red-hat-storage ocs-operator pull 1597 0 None open set upgrade conditions to true in external mode deployment 2022-03-23 06:35:39 UTC
Github red-hat-storage ocs-operator pull 1598 0 None Merged Bug 2066997: [release-4.9] set upgrade conditions in external mode deployment 2022-03-23 14:31:49 UTC
Red Hat Product Errata RHBA-2022:1237 0 None None None 2022-04-05 18:59:43 UTC

Comment 5 Vijay Avuthu 2022-03-25 10:52:50 UTC
Update:
========

tested below scenarios

1. upgrade from 4.9.5-4 to ocs-registry:4.10.0-210

Result: PASS
 	
 	https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster-prod/3629/console

2. upgrade from 4.8 live to 4.9.5-4

Result: PASS

	https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster-prod/3631/console

3. upgrade from 4.9 live to 4.9.5-4

Result: FAIL

    https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster-prod/3630/console

    must gather: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/vavuthuzup-z49fix/vavuthuzup-z49fix_20220325T075031/logs/failed_testcase_ocs_logs_1648197136/test_upgrade_ocs_logs/

$ oc get csv
NAME                  DISPLAY                       VERSION   REPLACES              PHASE
mcg-operator.v4.9.5   NooBaa Operator               4.9.5     mcg-operator.v4.9.4   Succeeded
ocs-operator.v4.9.4   OpenShift Container Storage   4.9.4     ocs-operator.v4.9.3   Replacing
ocs-operator.v4.9.5   OpenShift Container Storage   4.9.5     ocs-operator.v4.9.4   Pending
odf-operator.v4.9.5   OpenShift Data Foundation     4.9.5     odf-operator.v4.9.4   Succeeded

$ oc describe csv ocs-operator.v4.9.5
Name:         ocs-operator.v4.9.5
Namespace:    openshift-storage
Labels:       full_version=4.9.5-5

Status:
  Cleanup:
  Conditions:
    Last Transition Time:  2022-03-25T08:34:42Z
    Last Update Time:      2022-03-25T08:34:42Z
    Message:               requirements not yet checked
    Phase:                 Pending
    Reason:                RequirementsUnknown
    Last Transition Time:  2022-03-25T08:34:42Z
    Last Update Time:      2022-03-25T08:34:42Z
    Message:               operator is not upgradeable: The operator is not upgradeable: StorageCluster is not ready.
    Phase:                 Pending
    Reason:                OperatorConditionNotUpgradeable
  Last Transition Time:    2022-03-25T08:34:42Z
  Last Update Time:        2022-03-25T08:34:42Z
  Message:                 operator is not upgradeable: The operator is not upgradeable: StorageCluster is not ready.
  Phase:                   Pending
  Reason:                  OperatorConditionNotUpgradeable
Events:
  Type    Reason                           Age   From                        Message
  ----    ------                           ----  ----                        -------
  Normal  RequirementsUnknown              34m   operator-lifecycle-manager  requirements not yet checked
  Normal  OperatorConditionNotUpgradeable  34m   operator-lifecycle-manager  operator is not upgradeable: The operator is not upgradeable: StorageCluster is not ready.


$ oc describe storagecluster ocs-external-storagecluster
Name:         ocs-external-storagecluster
Namespace:    openshift-storage
Labels:       <none>
Annotations:  storagesystem.odf.openshift.io/watched-by: ocs-external-storagecluster-storagesystem
              uninstall.ocs.openshift.io/cleanup-policy: delete
              uninstall.ocs.openshift.io/mode: graceful

Last Heartbeat Time:   2022-03-25T09:09:43Z
    Last Transition Time:  2022-03-25T08:23:56Z
    Message:               Reconcile completed successfully
    Reason:                ReconcileCompleted
    Status:                True
    Type:                  Upgradeable
    Last Heartbeat Time:   2022-03-25T08:21:56Z
    Last Transition Time:  2022-03-25T08:21:51Z
    Message:               External CephCluster is trying to connect: Attempting to connect to an external Ceph cluster
    Reason:                ExternalClusterStateConnecting
    Status:                True
    Type:                  ExternalClusterConnecting
    Last Heartbeat Time:   2022-03-25T08:21:56Z
    Last Transition Time:  2022-03-25T08:21:51Z
    Message:               External CephCluster is trying to connect: Attempting to connect to an external Ceph cluster
    Reason:                ExternalClusterStateConnecting
    Status:                False
    Type:                  ExternalClusterConnected
  External Secret Hash:    247cd30c6e20f435bef14e171e828aa7394f45b8d7f4fd22fdf5d9772c2b7b45073b3a709296c8b0ef7aed18b4723139efba3d7ede6bd620966f57773282cd78
  Images:
    Ceph:
      Desired Image:  registry.redhat.io/rhceph/rhceph-5-rhel8@sha256:2296c19fbd3a0be84d6030dff789ce3e79b38cc30c39f45913aec97967b65cce
    Noobaa Core:
      Actual Image:   registry.redhat.io/odf4/mcg-core-rhel8@sha256:62c322de8d9818008b846778b2e2b51dede990c9cbf33cfe9dae23d744ec80d2
      Desired Image:  registry.redhat.io/odf4/mcg-core-rhel8@sha256:62c322de8d9818008b846778b2e2b51dede990c9cbf33cfe9dae23d744ec80d2
    Noobaa DB:
      Actual Image:   registry.redhat.io/rhel8/postgresql-12@sha256:b006af9af125adce28546d0ed9d77421abe54182741f6b5e1d31c88dd763580b
      Desired Image:  registry.redhat.io/rhel8/postgresql-12@sha256:b006af9af125adce28546d0ed9d77421abe54182741f6b5e1d31c88dd763580b
  Phase:              Ready
  Related Objects:
    API Version:       ceph.rook.io/v1
    Kind:              CephCluster
    Name:              ocs-external-storagecluster-cephcluster
    Namespace:         openshift-storage
    Resource Version:  56534
    UID:               d20d1481-8b86-47e7-98b9-15b5493982d0
    API Version:       noobaa.io/v1alpha1
    Kind:              NooBaa
    Name:              noobaa
    Namespace:         openshift-storage
    Resource Version:  56616
    UID:               59b49ac3-4775-4bf0-80b5-35db1a0c9893
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  CreationSucceeded  48m   StorageCluster controller  StorageSystem ocs-external-storagecluster-storagesystem created for the StorageCluster ocs-external-storagecluster.

$ oc get cephcluster
NAME                                      DATADIRHOSTPATH   MONCOUNT   AGE   PHASE       MESSAGE                          HEALTH      EXTERNAL
ocs-external-storagecluster-cephcluster                                49m   Connected   Cluster connected successfully   HEALTH_OK   true

Comment 11 errata-xmlrpc 2022-04-05 18:59:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Data Foundation 4.9.5 Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1237


Note You need to log in before you can comment on or make changes to this bug.