Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem (please be detailed as possible and provide log snippests): Upgrade from OCS 4.4 to 4.5 is failing to spin up 2 OSD pods: rook-ceph-osd-0-69447798c8-w6xsm 1/1 Running 0 14h rook-ceph-osd-1-7c5f79f5fc-dnv48 1/1 Running 0 14h rook-ceph-osd-2-79cc6bc7d9-hn9s8 1/1 Running 0 14h rook-ceph-osd-3-57677649fd-mlpvg 0/1 Pending 0 14h rook-ceph-osd-4-5c89c98f7c-j44s4 1/1 Running 0 14h rook-ceph-osd-5-6f78c894fc-57kfm 0/1 Pending 0 14h From describe of one of pod I see in the events: node.ocs.openshift.io/storage=true:NoSchedule Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling <unknown> default-scheduler 0/6 nodes are available: 6 Insufficient cpu. I see the CSV is in succeeded state: NAME DISPLAY VERSION REPLACES PHASE ocs-operator.v4.5.0-67.ci OpenShift Container Storage 4.5.0-67.ci ocs-operator.v4.4.2 Succeeded Ceph health shows also OK which is weird. oc rsh -n openshift-storage rook-ceph-tools-5f9d64c99b-cwx2c ceph health HEALTH_OK Version of all relevant components (if applicable): OCP 4.5.0-0.nightly-2020-09-08-123650 OCS 4.5 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Is there any workaround available to the best of your knowledge? Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? Can this issue reproducible? Yes Can this issue reproduce from the UI? Haven't tried yet If this is a regression, please provide more details to justify this: Steps to Reproduce: 1. Install OCS 4.4 on I3 LSO 2. Upgrade to OCS 4.5 3. Actual results: 2 OSDs pods are in pending state Expected results: Upgrade succeed Additional info: Job link: https://ocs4-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/qe-deploy-ocs-cluster/12117/ must-gather: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-ai3cl33-ua/jnk-ai3cl33-ua_20200908T220603/logs/failed_testcase_ocs_logs_1599606119/test_upgrade_ocs_logs/
0Note, When we install OCS 4.4, the Storagecluster CR is created with following limits (https://gist.github.com/mmgaggle/09fe2c51689c4dda7598042cc25ae526#file-ocs-4-4-storagecluster-yaml) But for OCS 4.5, one needs CPU limits to be set for OSDs, else all the Ceph pods would not come up . Also, in 4.5 recently, the noobaa pod limits were reverted to 4 GB and 1 CPU. Could this be the reason for CPU insufficient ? As we know, the Storage Cluster CR differs considerably between OCS 4.4 and OCS 4.5 for LSO (with limits added for OSDs and removed for noobaa pods). In the absence of limits for OSDs, could this be the reason that the existing OSDs are not UP post upgrade. We need to confirm how Upgrades are planned in the customer env, as it needs setting CPU limits to 1 for OSDs. Doc link for 4.5 - https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.5/html-single/deploying_openshift_container_storage_using_amazon_web_services/index?lb_target=preview#creating-openshift-container-storage-cluster-on-amazon-ec2_local-storage Expected OCS 4.5 CR (for fresh deployment) ----------------------------------------------- apiVersion: ocs.openshift.io/v1 kind: StorageCluster metadata: name: ocs-storagecluster namespace: openshift-storage spec: manageNodes: false resources: mds: limits: cpu: 3 memory: 8Gi requests: cpu: 1 memory: 8Gi monDataDirHostPath: /var/lib/rook storageDeviceSets: - count: 2 dataPVCTemplate: spec: accessModes: - ReadWriteOnce resources: requests: storage: 2328Gi storageClassName: localblock volumeMode: Block name: ocs-deviceset placement: {} portable: false replica: 3 resources: limits: cpu: 2 memory: 5Gi requests: cpu: 1 memory: 5Gi
(In reply to Neha Berry from comment #3) > 0Note, > > When we install OCS 4.4, the Storagecluster CR is created with following > limits > (https://gist.github.com/mmgaggle/09fe2c51689c4dda7598042cc25ae526#file-ocs- > 4-4-storagecluster-yaml) > > But for OCS 4.5, one needs CPU limits to be set for OSDs, else all the Ceph > pods would not come up . Also, in 4.5 recently, the noobaa pod limits were > reverted to 4 GB and 1 CPU. Could this be the reason for CPU insufficient ? Correct, both could be the reason. > > As we know, the Storage Cluster CR differs considerably between OCS 4.4 and > OCS 4.5 for LSO (with limits added for OSDs and removed for noobaa pods). > > In the absence of limits for OSDs, could this be the reason that the > existing OSDs are not UP post upgrade. We need to confirm how Upgrades are > planned in the customer env, as it needs setting CPU limits to 1 for OSDs. Correct! > > Doc link for 4.5 - > https://access.redhat.com/documentation/en-us/ > red_hat_openshift_container_storage/4.5/html-single/ > deploying_openshift_container_storage_using_amazon_web_services/ > index?lb_target=preview#creating-openshift-container-storage-cluster-on- > amazon-ec2_local-storage > > Expected OCS 4.5 CR (for fresh deployment) > ----------------------------------------------- > apiVersion: ocs.openshift.io/v1 > kind: StorageCluster > metadata: > name: ocs-storagecluster > namespace: openshift-storage > spec: > manageNodes: false > resources: > mds: > limits: > cpu: 3 > memory: 8Gi > requests: > cpu: 1 > memory: 8Gi > monDataDirHostPath: /var/lib/rook > storageDeviceSets: > - count: 2 > dataPVCTemplate: > spec: > accessModes: > - ReadWriteOnce > resources: > requests: > storage: 2328Gi > storageClassName: localblock > volumeMode: Block > name: ocs-deviceset > placement: {} > portable: false > replica: 3 > resources: > limits: > cpu: 2 > memory: 5Gi > requests: > cpu: 1 > memory: 5Gi Looks right. We are missing in 4.4 yaml is the OSD changes. The default Noobaa resources are lower than what we set for 4.4 so there is no point of providing them.
This requires a documentation fix.
@Orit, can you provide the required changes to the storagecluster CR (I assume that's what's needed) to be documented?