Bug 1859516 - OCS 4.5 AWS LSO: StorageCluster creation fails with "Invalid:Integer" & deployment fails even after adding the String to the CPU limits(Pending Pods)
Summary: OCS 4.5 AWS LSO: StorageCluster creation fails with "Invalid:Integer" & deplo...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Container Storage
Classification: Red Hat Storage
Component: ocs-operator
Version: 4.5
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: OCS 4.5.0
Assignee: umanga
QA Contact: Neha Berry
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-22 10:14 UTC by Petr Balogh
Modified: 2020-09-15 10:18 UTC (History)
10 users (show)

Fixed In Version: 4.5.0-518.ci
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-09-15 10:18:18 UTC
Embargoed:


Attachments (Terms of Use)
StorageClusterCr.yaml used for creating the Storage Cluster (784 bytes, text/plain)
2020-08-11 18:30 UTC, Neha Berry
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift ocs-operator pull 659 0 None closed use controller-gen v0.2.5 to fix validations for int-or-string types 2020-09-14 09:36:28 UTC
Github openshift ocs-operator pull 673 0 None closed Bug 1859516: x-int-or-string is not allowed for some fields 2020-09-14 09:36:28 UTC
Red Hat Product Errata RHBA-2020:3754 0 None None None 2020-09-15 10:18:47 UTC

Description Petr Balogh 2020-07-22 10:14:52 UTC
Description of problem (please be detailed as possible and provide log
snippests):
We hit this issue when verifying some change in our CI cause we hit the issue:
https://github.com/red-hat-storage/ocs-ci/issues/2551 .

https://github.com/red-hat-storage/ocs-ci/pull/2546#issuecomment-662360129

When applying storageCluster CR as mentioned in documentation we are hitting the issue:
Error is The StorageCluster "ocs-storagecluster" is invalid: 
E           * spec.resources.noobaa-endpoint.limits.cpu: Invalid value: "integer": spec.resources.noobaa-endpoint.limits.cpu in body must be of type string: "integer"
E           * spec.resources.noobaa-endpoint.requests.cpu: Invalid value: "integer": spec.resources.noobaa-endpoint.requests.cpu in body must be of type string: "integer"
E           * spec.resources.mds.limits.cpu: Invalid value: "integer": spec.resources.mds.limits.cpu in body must be of type string: "integer"
E           * spec.resources.mds.requests.cpu: Invalid value: "integer": spec.resources.mds.requests.cpu in body must be of type string: "integer"
E           * spec.resources.noobaa-core.limits.cpu: Invalid value: "integer": spec.resources.noobaa-core.limits.cpu in body must be of type string: "integer"
E           * spec.resources.noobaa-core.requests.cpu: Invalid value: "integer": spec.resources.noobaa-core.requests.cpu in body must be of type string: "integer"
E           * spec.resources.noobaa-db.limits.cpu: Invalid value: "integer": spec.resources.noobaa-db.limits.cpu in body must be of type string: "integer"
E           * spec.resources.noobaa-db.requests.cpu: Invalid value: "integer": spec.resources.noobaa-db.requests.cpu in body must be of type string: "integer"

So we are trying to transform CPU limit values to string in mentioned PR. But when verifying we see the error that two pods are in pending state:

http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr2546-b2386/jnk-pr2546-b2386_20200721T202041/logs/failed_testcase_ocs_logs_1595363204/test_deployment_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-9e7031bf262021cc63ccf37b7adac488e686469eff829c56f2ae60e14e7dcb94/oc_output/get_pods_-owide_-n_openshift-storage

rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-54cf68b8qcjvc   0/1     Pending     0          43m   <none>         <none>                                       <none>           <none>```

One of the pod: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr2546-b2386/jnk-pr2546-b2386_20200721T202041/logs/failed_testcase_ocs_logs_1595363204/test_deployment_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-9e7031bf262021cc63ccf37b7adac488e686469eff829c56f2ae60e14e7dcb94/ceph/namespaces/openshift-storage/pods/rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-54cf68b8qcjvc/rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-54cf68b8qcjvc.yaml

Comparing documentation:
https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.4/html-single/deploying_openshift_container_storage/index#creating-openshift-container-storage-cluster-on-amazon-ec2_aws-vmware

And our storageCluster CR:
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr2546-b2386/jnk-pr2546-b2386_20200721T202041/logs/failed_testcase_ocs_logs_1595363204/test_deployment_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-9e7031bf262021cc63ccf37b7adac488e686469eff829c56f2ae60e14e7dcb94/storagecluster.yaml

Even I don't see endpoint mentioned in doc preview for 4.5 - so I will open doc bz for that as well.
https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.5/html-single/deploying_openshift_container_storage/index?lb_target=preview#creating-openshift-container-storage-cluster-on-amazon-ec2_rhocs



I see: 
```status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-07-21T21:01:34Z"
    message: '0/6 nodes are available: 3 Insufficient cpu, 3 node(s) didn''t match
      node selector.'
For the nooba pod it looks the same:

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-07-21T21:04:46Z"
    message: '0/6 nodes are available: 3 Insufficient cpu, 3 node(s) had taint {node-role.kubernetes.io/master:
      }, that the pod didn't tolerate.'



Do we need to update documentation for 4.5 to contain this values to be in string format for cpu limits?   If so we will need to open the doc bug as well + I see we are missing noobaa-endpoint in 4.5 preview which is the new pod and I guess it should be part of those limits as well.


And anyway we see those pods are not coming up, so we don't have enough resources probably to handle the OCS 4.5 deployment on I3 atm.


Version of all relevant components (if applicable):
OCS: 4.5.0-494.ci
OCP: 4.5.0-0.nightly-2020-07-20-152128

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?
Yes


Can this issue reproduce from the UI?
Haven't tried but most likely it can be.


If this is a regression, please provide more details to justify this:
Worked in 4.4 and still working.


Steps to Reproduce:
1. Install OCS 4.5 on top of OCP 4.5 nightly on AWS LSO I3



Actual results:
2 pods are not starting with CPU limit

Expected results:
Have all pods running


Additional info:
Jenkins job:
https://ocs4-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/qe-deploy-ocs-cluster/10150/consoleFull

Must gather:
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-pr2546-b2386/jnk-pr2546-b2386_20200721T202041/logs/failed_testcase_ocs_logs_1595363204/test_deployment_ocs_logs/

Comment 3 Petr Balogh 2020-07-22 10:20:33 UTC
Documentation bug opened here:
https://bugzilla.redhat.com/show_bug.cgi?id=1859520

Comment 4 Sahina Bose 2020-07-27 08:06:51 UTC
Orit, did the default resource request/limit change in 4.5?
Do we need to update for AWS i3en instances?

Comment 5 Orit Wasserman 2020-07-27 09:42:26 UTC
Yes,the default resources were changed in 4.5:
OSD was moved to guaranteed QoS with equal requests and limits:  2 cpu and 5G memory
Noobaa resources were reduced: 
noobaa-core	1 cpu	4G
noobaa-db	0.5 cpu	0.5G
This means we should be able to deploy on i3en without any need to change the Noobaa resources.

Comment 6 Neha Berry 2020-07-27 10:22:33 UTC
(In reply to Orit Wasserman from comment #5)
> Yes,the default resources were changed in 4.5:
> OSD was moved to guaranteed QoS with equal requests and limits:  2 cpu and
> 5G memory
> Noobaa resources were reduced: 
> noobaa-core	1 cpu	4G
> noobaa-db	0.5 cpu	0.5G
> This means we should be able to deploy on i3en without any need to change
> the Noobaa resources.

@Orit thanks for confirming

We have raised Bug 1858283 to change steps to include StorageCluster install via UI . 

1. Does that mean in OCS 4.5, docs, we do not need to mention any resource limts/requests for any POD (currently, we have for noobaa-core, noobaa-db and MDS). See http://pastebin.test.redhat.com/887744

2. if yes to #1, then can we recommend to install StorageCluster from UI instead of the current documented CLI process ? 
  P.S: Install from UI will use only 1 of the 2 available disks . To bring up OSD on the 2nd disk, we would need to add capacity.

Please confirm.

Comment 7 Orit Wasserman 2020-07-27 12:27:57 UTC
(In reply to Neha Berry from comment #6)
> (In reply to Orit Wasserman from comment #5)
> > Yes,the default resources were changed in 4.5:
> > OSD was moved to guaranteed QoS with equal requests and limits:  2 cpu and
> > 5G memory
> > Noobaa resources were reduced: 
> > noobaa-core	1 cpu	4G
> > noobaa-db	0.5 cpu	0.5G
> > This means we should be able to deploy on i3en without any need to change
> > the Noobaa resources.
> 
> @Orit thanks for confirming
> 
> We have raised Bug 1858283 to change steps to include StorageCluster install
> via UI . 
> 
> 1. Does that mean in OCS 4.5, docs, we do not need to mention any resource
> limts/requests for any POD (currently, we have for noobaa-core, noobaa-db
> and MDS). See http://pastebin.test.redhat.com/887744

We still need to reduce the MDS requests to:
resources:
    mds:
      limits:
        cpu: 3
      requests:
        cpu: 1
For 4.6 we plan to make the MDS burstable by default(reduce requests cpu to 1).

> 
> 2. if yes to #1, then can we recommend to install StorageCluster from UI
> instead of the current documented CLI process ? 
>   P.S: Install from UI will use only 1 of the 2 available disks . To bring
> up OSD on the 2nd disk, we would need to add capacity.

Only with a single OSD, with two OSD we will exceed 8 cores (from my calculations)


> 
> Please confirm.

Comment 10 Petr Balogh 2020-07-29 12:42:43 UTC
Regards to integer change to string see reply here:
https://bugzilla.redhat.com/show_bug.cgi?id=1859386#c17

It looks like OCS issue which should be fixed.

Comment 16 Mudit Agarwal 2020-08-05 12:48:02 UTC
Backport PR merged: openshift/ocs-operator/pull/673

Comment 18 Neha Berry 2020-08-11 18:30:52 UTC
Created attachment 1711113 [details]
StorageClusterCr.yaml used for creating the Storage Cluster

Verified the fix and installation succeeds even without using quotes for CPU and Memory Limits. Hence , moving the BZ to verified state.

>> Versions

$ oc get csv
NAME                         DISPLAY                       VERSION        REPLACES   PHASE
ocs-operator.v4.5.0-521.ci   OpenShift Container Storage   4.5.0-521.ci              Installing

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2020-08-10-150345   True        False         24m     Cluster version is 4.5.0-0.nightly-2020-08-10-150345

>> Hardware

$ oc get machines -A
NAMESPACE               NAME                                            PHASE     TYPE           REGION      ZONE         AGE
openshift-machine-api   nberry-aws-lso4-l5ppg-master-0                  Running   m4.xlarge      us-east-2   us-east-2a   31m
openshift-machine-api   nberry-aws-lso4-l5ppg-master-1                  Running   m4.xlarge      us-east-2   us-east-2b   31m
openshift-machine-api   nberry-aws-lso4-l5ppg-master-2                  Running   m4.xlarge      us-east-2   us-east-2c   31m
openshift-machine-api   nberry-aws-lso4-l5ppg-worker-us-east-2a-c8sbh   Running   i3en.2xlarge   us-east-2   us-east-2a   18m
openshift-machine-api   nberry-aws-lso4-l5ppg-worker-us-east-2b-wvlwv   Running   i3en.2xlarge   us-east-2   us-east-2b   18m
openshift-machine-api   nberry-aws-lso4-l5ppg-worker-us-east-2c-c76ln   Running   i3en.2xlarge   us-east-2   us-east-2c   18m

________________________________________________________________________________


Steps Performed with output:
-----------------------------
>> a. labelled the nodes

$ for i in $(oc get nodes|grep worker|awk '{print$1}');  do echo $i;   oc label node $i cluster.ocs.openshift.io/openshift-storage=''; done


>> b. $ oc get nodes -l cluster.ocs.openshift.io/openshift-storage=
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-0-130-229.us-east-2.compute.internal   Ready    worker   18m   v1.18.3+002a51f
ip-10-0-177-229.us-east-2.compute.internal   Ready    worker   18m   v1.18.3+002a51f
ip-10-0-196-54.us-east-2.compute.internal    Ready    worker   18m   v1.18.3+002a51f

>> c. Created Namespace "local-storage". Installed Local Storage Operator  from UI (Stable-4.5)

local-storage-operator-69bdfc86bf-qphfk   1/1     Running   0          18m   10.129.2.11   ip-10-0-177-229.us-east-2.compute.internal   <none>           <none>


>> d.  available storage devices

>>$for i in $(oc get nodes|grep worker|awk '{print$1}');  do echo $i;   oc debug node/$i -- chroot /host /bin/bash -c "ls -l /dev/disk/by-id/|grep Storage" ; done

ip-10-0-130-229.us-east-2.compute.internal
Starting pod/ip-10-0-130-229us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
lrwxrwxrwx. 1 root root 13 Aug 11 17:44 nvme-Amazon_EC2_NVMe_Instance_Storage_AWS274E3DED76363339E -> ../../nvme1n1
lrwxrwxrwx. 1 root root 13 Aug 11 17:44 nvme-Amazon_EC2_NVMe_Instance_Storage_AWS67448B4FE57ADCE1B -> ../../nvme2n1

Removing debug pod ...
ip-10-0-177-229.us-east-2.compute.internal
Starting pod/ip-10-0-177-229us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
lrwxrwxrwx. 1 root root 13 Aug 11 17:44 nvme-Amazon_EC2_NVMe_Instance_Storage_AWS1726229B65254078C -> ../../nvme1n1
lrwxrwxrwx. 1 root root 13 Aug 11 17:44 nvme-Amazon_EC2_NVMe_Instance_Storage_AWS27B1DA5D6B23E86C5 -> ../../nvme2n1

Removing debug pod ...
ip-10-0-196-54.us-east-2.compute.internal
Starting pod/ip-10-0-196-54us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
lrwxrwxrwx. 1 root root 13 Aug 11 17:45 nvme-Amazon_EC2_NVMe_Instance_Storage_AWS2780231DF85A08CC9 -> ../../nvme1n1
lrwxrwxrwx. 1 root root 13 Aug 11 17:45 nvme-Amazon_EC2_NVMe_Instance_Storage_AWS61D6CC35952F5387D -> ../../nvme2n1


>> e. Since localVolumeCr also had toleration for taint, to test deployment with tainted nodes, tainted all 3 Worker nodes with OCS taint

for i in $(oc get nodes|grep worker|awk '{print$1}');  do echo $i; oc adm taint node $i  node.ocs.openshift.io/storage=true:NoSchedule; done


>> f. Created localvolumeCr.yaml with the exact content from the docs [1] 

$ oc create -f locavolumeCr.yaml 
localvolume.local.storage.openshift.io/local-block created


$ oc -n local-storage get pods
NAME                                      READY   STATUS    RESTARTS   AGE
local-block-local-diskmaker-2flf2         1/1     Running   0          30s
local-block-local-diskmaker-9hm2s         1/1     Running   0          30s
local-block-local-diskmaker-s6dgx         1/1     Running   0          30s
local-block-local-provisioner-nhx9x       1/1     Running   0          30s
local-block-local-provisioner-tb49r       1/1     Running   0          30s
local-block-local-provisioner-v24d4       1/1     Running   0          30s
local-storage-operator-69bdfc86bf-qphfk   1/1     Running   0          8m25s
[nberry@localhost aug11]$ oc get pod local-block-local-diskmaker-2flf2 -o yaml -n local-storage|grep -i toler -A2 -B2
        f:serviceAccountName: {}
        f:terminationGracePeriodSeconds: {}
        f:tolerations: {}
        f:volumes:
          .: {}
--
  serviceAccountName: local-storage-admin
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoSchedule
    key: node.ocs.openshift.io/storage
[nberry@localhost aug11]$ 
[nberry@localhost aug11]$ 
[nberry@localhost aug11]$ oc get pv
NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
local-pv-3d915da1   2328Gi     RWO            Delete           Available           localblock              44s
local-pv-6d8cd8e8   2328Gi     RWO            Delete           Available           localblock              40s
local-pv-73a45a9a   2328Gi     RWO            Delete           Available           localblock              44s
local-pv-a5be9faa   2328Gi     RWO            Delete           Available           localblock              50s
local-pv-c002431f   2328Gi     RWO            Delete           Available           localblock              50s
local-pv-e3c92f9d   2328Gi     RWO            Delete           Available           localblock              40s
[nberry@localhost aug11]$ oc get sc | grep localblock
localblock      kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  79s

>>g. Installed OCS Operator from UI
========CSV ======
NAME                         DISPLAY                       VERSION        REPLACES   PHASE
ocs-operator.v4.5.0-521.ci   OpenShift Container Storage   4.5.0-521.ci              Succeeded

>>h. Created the StorageclusterCr without using quotes

$ date --utc; oc create -f StorageClusterCr.yaml 
Tue Aug 11 18:18:37 UTC 2020



>> Installation verification

--------------
========CSV ======
NAME                         DISPLAY                       VERSION        REPLACES   PHASE
ocs-operator.v4.5.0-521.ci   OpenShift Container Storage   4.5.0-521.ci              Succeeded
--------------
=======PODS ======
NAME                                                              READY   STATUS      RESTARTS   AGE     IP             NODE                                         NOMINATED NODE   READINESS GATES
csi-cephfsplugin-b8k6f                                            3/3     Running     0          8m41s   10.0.130.229   ip-10-0-130-229.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-k68b4                                            3/3     Running     0          8m40s   10.0.196.54    ip-10-0-196-54.us-east-2.compute.internal    <none>           <none>
csi-cephfsplugin-provisioner-cbbd44b-2c7n9                        5/5     Running     0          8m40s   10.129.2.16    ip-10-0-177-229.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-provisioner-cbbd44b-592l5                        5/5     Running     0          8m40s   10.129.2.15    ip-10-0-177-229.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-wl495                                            3/3     Running     0          8m40s   10.0.177.229   ip-10-0-177-229.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-bknf2                                               3/3     Running     0          8m41s   10.0.177.229   ip-10-0-177-229.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-provisioner-848b585c9c-rjmwl                        5/5     Running     0          8m41s   10.131.0.21    ip-10-0-196-54.us-east-2.compute.internal    <none>           <none>
csi-rbdplugin-provisioner-848b585c9c-xbdkw                        5/5     Running     0          8m41s   10.128.2.14    ip-10-0-130-229.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-qsg56                                               3/3     Running     0          8m41s   10.0.196.54    ip-10-0-196-54.us-east-2.compute.internal    <none>           <none>
csi-rbdplugin-vzdn7                                               3/3     Running     0          8m41s   10.0.130.229   ip-10-0-130-229.us-east-2.compute.internal   <none>           <none>
noobaa-core-0                                                     1/1     Running     0          6m16s   10.131.0.30    ip-10-0-196-54.us-east-2.compute.internal    <none>           <none>
noobaa-db-0                                                       1/1     Running     0          6m16s   10.128.2.24    ip-10-0-130-229.us-east-2.compute.internal   <none>           <none>
noobaa-endpoint-fbbd59bb8-vc48d                                   1/1     Running     0          3m40s   10.128.2.25    ip-10-0-130-229.us-east-2.compute.internal   <none>           <none>
noobaa-operator-6cbc8b77cc-g4z4b                                  1/1     Running     0          11m     10.128.2.13    ip-10-0-130-229.us-east-2.compute.internal   <none>           <none>
ocs-operator-6797b6f99d-hpbp8                                     1/1     Running     0          11m     10.131.0.20    ip-10-0-196-54.us-east-2.compute.internal    <none>           <none>
rook-ceph-crashcollector-ip-10-0-130-229-64ccd7c974-pgj6q         1/1     Running     0          7m25s   10.128.2.17    ip-10-0-130-229.us-east-2.compute.internal   <none>           <none>
rook-ceph-crashcollector-ip-10-0-177-229-6ff4b788c5-vgnzd         1/1     Running     0          7m34s   10.129.2.22    ip-10-0-177-229.us-east-2.compute.internal   <none>           <none>
rook-ceph-crashcollector-ip-10-0-196-54-7c4c94dccb-ht7ll          1/1     Running     0          7m10s   10.131.0.24    ip-10-0-196-54.us-east-2.compute.internal    <none>           <none>
rook-ceph-drain-canary-3424a0919b14999fa9209525288383db-5c2qjpl   1/1     Running     0          6m20s   10.128.2.21    ip-10-0-130-229.us-east-2.compute.internal   <none>           <none>
rook-ceph-drain-canary-398478a1d942ecad5c45659a3bc49352-6dg89k5   1/1     Running     0          6m24s   10.131.0.28    ip-10-0-196-54.us-east-2.compute.internal    <none>           <none>
rook-ceph-drain-canary-7dc1622ed028983a372236b0177712a0-84qmblb   1/1     Running     0          6m27s   10.129.2.23    ip-10-0-177-229.us-east-2.compute.internal   <none>           <none>
rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-9646d4f6r9dtm   1/1     Running     0          6m6s    10.131.0.31    ip-10-0-196-54.us-east-2.compute.internal    <none>           <none>
rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-b868c9fb782wc   1/1     Running     0          6m4s    10.128.2.23    ip-10-0-130-229.us-east-2.compute.internal   <none>           <none>
rook-ceph-mgr-a-846b8969bc-dl2lh                                  1/1     Running     0          6m54s   10.129.2.19    ip-10-0-177-229.us-east-2.compute.internal   <none>           <none>
rook-ceph-mon-a-5b4b9bbb5f-h4vz6                                  1/1     Running     0          7m35s   10.129.2.18    ip-10-0-177-229.us-east-2.compute.internal   <none>           <none>
rook-ceph-mon-b-5846cb5c74-sjk7v                                  1/1     Running     0          7m25s   10.128.2.16    ip-10-0-130-229.us-east-2.compute.internal   <none>           <none>
rook-ceph-mon-c-7749976b9b-frhv8                                  1/1     Running     0          7m10s   10.131.0.23    ip-10-0-196-54.us-east-2.compute.internal    <none>           <none>
rook-ceph-operator-577549c4c8-qn2nk                               1/1     Running     0          11m     10.129.2.14    ip-10-0-177-229.us-east-2.compute.internal   <none>           <none>
rook-ceph-osd-0-5846c5bdbf-hjdp8                                  1/1     Running     0          6m27s   10.129.2.24    ip-10-0-177-229.us-east-2.compute.internal   <none>           <none>
rook-ceph-osd-1-5cf57b7d6f-n6xkn                                  1/1     Running     0          6m25s   10.131.0.27    ip-10-0-196-54.us-east-2.compute.internal    <none>           <none>
rook-ceph-osd-2-7c7dfb8d8c-7wlnv                                  1/1     Running     0          6m23s   10.131.0.29    ip-10-0-196-54.us-east-2.compute.internal    <none>           <none>
rook-ceph-osd-3-744df45759-xl26l                                  1/1     Running     0          6m26s   10.129.2.25    ip-10-0-177-229.us-east-2.compute.internal   <none>           <none>
rook-ceph-osd-4-84f8bf568b-2t5n2                                  1/1     Running     0          6m21s   10.128.2.20    ip-10-0-130-229.us-east-2.compute.internal   <none>           <none>
rook-ceph-osd-5-7d68fb6d7d-772td                                  1/1     Running     0          6m18s   10.128.2.22    ip-10-0-130-229.us-east-2.compute.internal   <none>           <none>
rook-ceph-osd-prepare-ocs-deviceset-0-data-0-2kwmr-478t8          0/1     Completed   0          6m47s   10.129.2.20    ip-10-0-177-229.us-east-2.compute.internal   <none>           <none>
rook-ceph-osd-prepare-ocs-deviceset-0-data-1-9ls6t-sbfwc          0/1     Completed   0          6m46s   10.129.2.21    ip-10-0-177-229.us-east-2.compute.internal   <none>           <none>
rook-ceph-osd-prepare-ocs-deviceset-1-data-0-486vk-kpmjr          0/1     Completed   0          6m46s   10.131.0.25    ip-10-0-196-54.us-east-2.compute.internal    <none>           <none>
rook-ceph-osd-prepare-ocs-deviceset-1-data-1-kpm9g-rc97k          0/1     Completed   0          6m45s   10.131.0.26    ip-10-0-196-54.us-east-2.compute.internal    <none>           <none>
rook-ceph-osd-prepare-ocs-deviceset-2-data-0-kwh9p-qrldp          0/1     Completed   0          6m45s   10.128.2.18    ip-10-0-130-229.us-east-2.compute.internal   <none>           <none>
rook-ceph-osd-prepare-ocs-deviceset-2-data-1-wxrjc-c9jvj          0/1     Completed   0          6m44s   10.128.2.19    ip-10-0-130-229.us-east-2.compute.internal   <none>           <none>
--------------
======= PVC ==========
NAME                           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
db-noobaa-db-0                 Bound    pvc-b880c162-4996-465e-962d-d56bd4f30f4c   50Gi       RWO            ocs-storagecluster-ceph-rbd   6m17s
ocs-deviceset-0-data-0-2kwmr   Bound    local-pv-a5be9faa                          2328Gi     RWO            localblock                    6m52s
ocs-deviceset-0-data-1-9ls6t   Bound    local-pv-c002431f                          2328Gi     RWO            localblock                    6m52s
ocs-deviceset-1-data-0-486vk   Bound    local-pv-3d915da1                          2328Gi     RWO            localblock                    6m51s
ocs-deviceset-1-data-1-kpm9g   Bound    local-pv-73a45a9a                          2328Gi     RWO            localblock                    6m50s
ocs-deviceset-2-data-0-kwh9p   Bound    local-pv-e3c92f9d                          2328Gi     RWO            localblock                    6m49s
ocs-deviceset-2-data-1-wxrjc   Bound    local-pv-6d8cd8e8                          2328Gi     RWO            localblock                    6m49s
--------------
======= storagecluster ==========
NAME                 AGE     PHASE   EXTERNAL   CREATED AT             VERSION
ocs-storagecluster   8m50s   Ready              2020-08-11T18:18:38Z   4.5.0
--------------
======= cephcluster ==========
NAME                             DATADIRHOSTPATH   MONCOUNT   AGE     PHASE   MESSAGE                        HEALTH
ocs-storagecluster-cephcluster   /var/lib/rook     3          8m51s   Ready   Cluster created successfully   HEALTH_OK
======= backingstore ==========
NAME                           TYPE     PHASE   AGE
noobaa-default-backing-store   aws-s3   Ready   3m44s
======= PV ====
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                            STORAGECLASS                  REASON   AGE
local-pv-3d915da1                          2328Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-1-data-0-486vk   localblock                             15m
local-pv-6d8cd8e8                          2328Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-2-data-1-wxrjc   localblock                             15m
local-pv-73a45a9a                          2328Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-1-data-1-kpm9g   localblock                             15m
local-pv-a5be9faa                          2328Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-0-data-0-2kwmr   localblock                             15m
local-pv-c002431f                          2328Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-0-data-1-9ls6t   localblock                             15m
local-pv-e3c92f9d                          2328Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-2-data-0-kwh9p   localblock                             15m
pvc-b880c162-4996-465e-962d-d56bd4f30f4c   50Gi       RWO            Delete           Bound    openshift-storage/db-noobaa-db-0                 ocs-storagecluster-ceph-rbd            6m1s
______________________________________________________________________________


[1]  https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.5/html-single/deploying_openshift_container_storage/index?lb_target=preview#creating-openshift-container-storage-cluster-on-amazon-ec2_aws-vmware

Comment 21 errata-xmlrpc 2020-09-15 10:18:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Container Storage 4.5.0 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3754


Note You need to log in before you can comment on or make changes to this bug.