Here's my observation:

This is the StorageCluster status

```
  status:
    conditions:
    - lastHeartbeatTime: "2021-08-20T05:33:41Z"
      lastTransitionTime: "2021-08-20T05:33:40Z"
      message: CephCluster resource is not reporting status
      reason: CephClusterStatus
      status: "False"
      type: Available
    - lastHeartbeatTime: "2021-08-20T06:38:34Z"
      lastTransitionTime: "2021-08-20T05:33:40Z"
      message: Waiting on Nooba instance to finish initialization
      reason: NoobaaInitializing
      status: "True"
      type: Progressing
    - lastHeartbeatTime: "2021-08-20T05:37:38Z"
      lastTransitionTime: "2021-08-20T05:33:41Z"
      message: 'CephCluster is creating: Processing OSD 2 on PVC "ocs-deviceset-0-data-0jl5s5"'
      reason: ClusterStateCreating
      status: "False"
      type: Upgradeable
```

It says CephCluster is NotReady because it is still processing OSD 2. And, NooBaa is still not initialized.

Issue with CephCluster status is most probably due to failure in status update. Because, Ceph health seems to be fine with all OSDs accounted for.
And if it was actually failing, my experience is that NooBaa failure will call it out explicitly (which it didn't in this case).

This is CephCluster state

```
status:
  ceph:
    capacity:
      bytesAvailable: 319240495104
      bytesTotal: 322122547200
      bytesUsed: 2882052096
      lastUpdated: "2021-08-20T06:41:31Z"
    health: HEALTH_OK
    lastChanged: "2021-08-20T05:38:40Z"
    lastChecked: "2021-08-20T06:41:31Z"
    previousHealth: HEALTH_WARN
  message: Cluster created successfully
  phase: Ready
  state: Created
```

Now, this is NooBaa status

```
status:
  conditions:
  - lastHeartbeatTime: "2021-08-20T05:37:38Z"
    lastTransitionTime: "2021-08-20T05:37:38Z"
    message: |-
      RequestError: send request failed
      caused by: Put "https://rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage.svc:443/nb.1629441716656.apps.vavuthu6-ocs49.qe.rh-ocs.com": x509: certificate signed by unknown authority
    reason: TemporaryError
    status: "False"
    type: Available
```

It says it's hitting certificate issue which is probably where NooBaa operator is stuck.

So this marks StorageCluster as Progressing. Which means OCS Operator pod is NotReady (i.e. it's still hitting errors). So CSV is also stuck in installing.

We need to figure 2 things here:
1. Why are we hitting certs issue? Did anything change here?
2. Why is the status mixed up? It still might not be reflecting the actual state or maybe that was the state at log collection time.

On IBM Power platform (ppc64le) , we are also hitting the same issue. Tried both with UI and ocs-ci, issue is same in both the cases. 

CSV: 

[root@rdr-aar49-sao01-bastion-0 ~]# oc get csv -n openshift-storage
NAME                         DISPLAY                       VERSION        REPLACES   PHASE
ocs-operator.v4.9.0-102.ci   OpenShift Container Storage   4.9.0-102.ci              Installing


Pods : 

[root@rdr-aar49-sao01-bastion-0 ~]# oc get pods -n openshift-storage
NAME                                                              READY   STATUS      RESTARTS   AGE
csi-cephfsplugin-6cdhn                                            3/3     Running     0          3d3h
csi-cephfsplugin-provisioner-54fbb98c8f-9jf76                     6/6     Running     0          3d3h
csi-cephfsplugin-provisioner-54fbb98c8f-wz9gc                     6/6     Running     0          3d3h
csi-cephfsplugin-scdzd                                            3/3     Running     0          3d3h
csi-cephfsplugin-wdn92                                            3/3     Running     0          3d3h
csi-rbdplugin-5xrpd                                               3/3     Running     0          3d3h
csi-rbdplugin-8gqjn                                               3/3     Running     0          3d3h
csi-rbdplugin-provisioner-84ccc64b48-7xqzw                        6/6     Running     0          3d3h
csi-rbdplugin-provisioner-84ccc64b48-pmh9h                        6/6     Running     0          3d3h
csi-rbdplugin-pwvxx                                               3/3     Running     0          3d3h
noobaa-core-0                                                     1/1     Running     0          3d3h
noobaa-db-pg-0                                                    1/1     Running     0          3d3h
noobaa-endpoint-6f854c9848-9c9s7                                  1/1     Running     0          3d3h
noobaa-operator-68998c44dc-mtqcf                                  1/1     Running     0          3d3h
ocs-metrics-exporter-7455f88587-tq56g                             1/1     Running     0          3d3h
ocs-operator-7d8bb7577d-tnf8d                                     0/1     Running     0          3d3h
rook-ceph-crashcollector-rdr-aar49-sao01-worker-0-59ffb784crr7k   1/1     Running     0          3d3h
rook-ceph-crashcollector-rdr-aar49-sao01-worker-1-5bd7bdbbnktmk   1/1     Running     0          3d3h
rook-ceph-crashcollector-rdr-aar49-sao01-worker-2-89fd967c6db9h   1/1     Running     0          3d3h
rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-577fd545kl7k7   2/2     Running     0          3d3h
rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-695f47b4gr2pl   2/2     Running     0          3d3h
rook-ceph-mgr-a-668c4fc9b6-75l7m                                  2/2     Running     0          3d3h
rook-ceph-mon-a-55f545b46d-z42np                                  2/2     Running     0          3d3h
rook-ceph-mon-b-689568457b-s59xc                                  2/2     Running     0          3d3h
rook-ceph-mon-c-796f6d449c-cpvfj                                  2/2     Running     0          3d3h
rook-ceph-operator-5c6c56b95-kbtlc                                1/1     Running     0          3d3h
rook-ceph-osd-0-7ddc744477-bxqcb                                  2/2     Running     0          3d3h
rook-ceph-osd-1-b6b98d8b-k8dkm                                    2/2     Running     0          3d3h
rook-ceph-osd-2-5487f7c89f-ps97v                                  2/2     Running     0          3d3h
rook-ceph-osd-prepare-ocs-deviceset-localblock-0-data---1-d2ql7   0/1     Completed   0          3d3h
rook-ceph-osd-prepare-ocs-deviceset-localblock-0-data---1-qsgld   0/1     Completed   0          3d3h
rook-ceph-osd-prepare-ocs-deviceset-localblock-0-data---1-txqww   0/1     Completed   0          3d3h
rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-595cd6blt94s   2/2     Running     0          3d3h
rook-ceph-tools-cdd8d5c65-lfrq5                                   1/1     Running     0          3d3h


events of ocs-operator pod : 

Events:
  Type     Reason      Age                       From     Message
  ----     ------      ----                      ----     -------
  Warning  ProbeError  2m48s (x30681 over 3d3h)  kubelet  Readiness probe error: HTTP probe failed with statuscode: 500
body: [-]readyz failed: reason withheld
healthz check failed


events of ocs-operator CSV : 

Events:
  Type     Reason              Age                    From                        Message
  ----     ------              ----                   ----                        -------
  Warning  InstallCheckFailed  42s (x1761 over 3d3h)  operator-lifecycle-manager  install timeout

must-gather logs for ODF4.9 installation on ppc64le : https://drive.google.com/file/d/1d8ZAn-vtY8j4YI6wJIFJDbnc2AExqQYa/view?usp=sharing

Is there any live set up available, in 4.9 for RGW we opened up SSL port with help of the serving cert feature(till now only insecure port 80 was enabled)

in noobaa-operator we are trying to create a bucket in RGW here: https://github.com/noobaa/noobaa-operator/blob/e9400e06fbb438b52ba0946e17e76a13f16b0b1f/pkg/system/phase4_configuring.go#L980-L998

we use the endpoint provided in object-store-user secret.

cluster is in same state.

Kubeconfig: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/vavuthu6-ocs49/vavuthu6-ocs49_20210820T044517/openshift-cluster-dir/auth/kubeconfig

(In reply to Vijay Avuthu from comment #8)
> cluster is in same state.
> 
> Kubeconfig:
> http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/vavuthu6-
> ocs49/vavuthu6-ocs49_20210820T044517/openshift-cluster-dir/auth/kubeconfig

Thanks for sharing the cluster, I checked the state of cephobjectstore and cephobjectstoreuser it looks good to me.

As Nirmod mentioned in https://chat.google.com/room/AAAAREGEba8/PJsC-wuS_98, the secure port was opened for RGW from 4.9 onwards, so we need changes in Noobaa to accommodate the same.

For internal mode, Openshift serving cert feature is used, so the cert should be available in each pod AFAIR.

For the external mode of the ceph cluster, since is ceph cluster is already configured and most probably running outside of openshift it is expected them to provide the TLS certs as secrets for RGW.

*** Bug 1998065 has been marked as a duplicate of this bug. ***

We are (IBM Power platform) is still seeing the problem

ocs-operator.v4.9.0-117.ci is not in expected phase: Succeeded

the csv continues to be in installing state.

Mudit asked me to collet must-gather. But found another issue : https://bugzilla.redhat.com/show_bug.cgi?id=1999267
ocs-must-gather is available in quay.io/rhcephdev only for x86 platform.

The root cause for ODF 4.9 deploy failure for system P is missing ppc64le image for ocs-operator. (also ocs-must-gather as noted above). Both ocs-operator and ocs-must-gather are getting built only for x86.

Please don't overload with different issues on the same BZ

For the issue Sridhar mentioned we have https://bugzilla.redhat.com/show_bug.cgi?id=1999267

For the original issue there is another workaround in place and we will get it with the next build.

I can confirm that this issue has been solved in ODF operator build 4.9.0-120.ci on the s390x platform. There is however a new issue that prevents the ODF operator from installing properly on s390x. I will create a new ticket for that one.

@muagarwa is this already fixed? If so, this bug should be moved to ON_QE please.

Thanks

AFAIK, its not fixed completely. A workaround is there in place by the Noobaa team.
Nimrod, can confirm.

Elad, the workaround (its a code fix) is working and we are no longer blocked. 
I can't remove the blocker flag because it has a keyword 'regression', is it fine to move this BZ to ON_QA and track the complete fix with a new bz?

Verified with build 4.9.0-194.ci

> All operators are in succeeded state

NAME                     DISPLAY                       VERSION   REPLACES   PHASE
noobaa-operator.v4.9.0   NooBaa Operator               4.9.0                Succeeded
ocs-operator.v4.9.0      OpenShift Container Storage   4.9.0                Succeeded
odf-operator.v4.9.0      OpenShift Data Foundation     4.9.0                Succeeded

Job: https://ocs4-jenkins-csb-ocsqe.apps.ocp4.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/6911/consoleFull

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat OpenShift Data Foundation 4.9.0 enhancement, security, and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:5085