Description of problem (please be detailed as possible and provide log snippests): -------------------------------------------------------------------------- This Bug is raised to track the downstream fix for the Bug 1782872 to "Restart portable osd pod if stuck." Till now, we had documented manual steps for recovering an OSD on a different node if its hosting node goes to NotRead state (see Bug 1782872#c19) But, now PR -https://github.com/rook/rook/pull/5376 is Up for fixing this manual intervention Some background from above bug ----------------------------------- Consider the following scenario: - MON, OSD and app pods (using RWO PVC) were running on the NODE1. - The NODE1 becomes unresponsive, then consequently goes into NotReady state. - Now the pods will try to come up on NODE2 but they will get stuck due to Multi-Attach error. After this, to bring up the OSD on the NODE2, we had been recommending force deletion of the pod from Node1. . - On force deletion of pods (which will be in Terminating state) on NODE1, the pods on NODE2 will come into Running state and will be able to use the pvc Version of all relevant components (if applicable): -------------------------------------------------------------------------- OCS 4.2 , OCS 4.3 and OCS 4.4 This is the general behavior seen for RBD Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? -------------------------------------------------------------------------- Yes Is there any workaround available to the best of your knowledge? -------------------------------------------------------------------------- Ye smanual workaround Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? -------------------------------------------------------------------------- 4 Can this issue reproducible? -------------------------------------------------------------------------- Yes Can this issue reproduce from the UI? -------------------------------------------------------------------------- Yes If this is a regression, please provide more details to justify this: -------------------------------------------------------------------------- No Steps to Reproduce: -------------------------------------------------------------------------- 1. Make one of the worker node NODE1 unresponsive so as to make it NotReady 2. After few minutes the Mon, OSD and app-pods (using RWO PVC) will try to come up on NODE2 but they will get stuck due to Multi-Attach error message Note: Mon will get stuck in Init when NODE2 is the extra OCS labelled node in the same zone as that of NODE1 3. OSD should move to the new node and come into Running state. Actual results: -------------------------------------------------------------------------- OSD pods will try to come up on NODE2 but they will get stuck due to Multi-Attach error Expected results: -------------------------------------------------------------------------- OSD pod should come up on NODE2 without manual intervention of force delete of the pods Additional info: -------------------------------------------------------------------------- When 1 node is in NotReady state, the following states of pods are observed: - Mon and OSD pod on NODE2 will be stuck in Init:0/2 and Init:0/3 respectively due to Multi-Attach error - App-pods on NODE2 will be stuck in ContainerCreating due to Multi-Attach error rook-ceph-mon-g-79c8d6bd86-r4958 0/1 Init:0/2 0 12m <none> ip-10-0-150-124.us-east-2.compute.internal <none> <none> rook-ceph-mon-g-79c8d6bd86-tfbnl 1/1 Terminating 0 70m 10.129.2.53 ip-10-0-145-3.us-east-2.compute.internal <none> <none> rook-ceph-osd-2-6559c54db4-5s9cp 0/1 Init:0/3 0 12m <none> ip-10-0-150-124.us-east-2.compute.internal <none> rook-ceph-osd-2-6559c54db4-k8qfr 1/1 Terminating 0 3h11m 10.129.2.26 ip-10-0-145-3.us-east-2.compute.internal <none> <none> cephfspod-1-2zwjf 1/1 Terminating 0 27m 10.129.2.30 ip-10-0-145-3.us-east-2.compute.internal <none> <none> cephfspod-1-7bj9h 0/1 ContainerCreating 0 14m <none> ip-10-0-140-31.us-east-2.compute.internal <none> <none>
Upstream PR is being tested for this scenario: https://github.com/rook/rook/pull/5376 The fix is scoped and low risk IMO. I would propose this as a 4.4 blocker if we can show that it improves the IBM failure scenario.
Local and unit testing is working. The operator will delete the pod if all these conditions are met:: 1. If an OSD is "down" and "out" 2. If the OSD pod is marked as deleted (has a deletionTimestamp) 3. If the node where the OSD was assigned is not in "ready" state (it will be in "unknown" state in the expected scenario) A test image has been pushed where others can also test it. The Rook operator deployment needs to be updated to the image "travisn/ceph:delete-stuck-osd" in order to test that the operator will force delete the osd deployments that are stuck.
tested on AWS cluster. 6 instances. I had 3 nodes for rook/ceph, making the rest available. on first trial, I've used "halt -p" to shutdown a node that had an OSD on it (and other rook/ceph pods as well). the second the node went down, a new OSD was created and got stuck at "Init:0/4" waiting for the gp2 volume. the old OSD remains with status "Terminating". Ceph showed the OSD as "down". the rook operator checks for "down" OSDs every 5 minutes, so there's a little bit of timing here, so after something like 13 minutes, I can see the patch removing the "old" OSD: 2020-05-01 00:36:45.337952 I | op-osd: validating status of osd.0 2020-05-01 00:36:45.337972 I | op-osd: osd.0 is marked 'DOWN' 2020-05-01 00:36:45.337987 I | op-osd: osd.0 is marked 'OUT' 2020-05-01 00:36:45.348218 I | op-osd: checking if osd 0 pod is stuck and should be force deleted 2020-05-01 00:36:45.348232 I | op-osd: skipping restart of OSD 0 since the pod is not deleted 2020-05-01 00:36:45.348236 I | op-osd: checking if osd 0 pod is stuck and should be force deleted 2020-05-01 00:36:45.350226 I | op-osd: force deleting pod "rook-ceph-osd-0-6d6b4ddf96-sh5qn" that appears to be stuck terminating 2020-05-01 00:36:45.359110 I | op-osd: pod "rook-ceph-osd-0-6d6b4ddf96-sh5qn" deletion succeeded looking at describe of the new osd pod, we see it is still waiting: Normal Scheduled <unknown> default-scheduler Successfully assigned rook-ceph/rook-ceph-osd-0-6d6b4ddf96-jdzmt to ip-10-0-142-177.us-west-2.compute.internal Warning FailedAttachVolume 15m attachdetach-controller Multi-Attach error for volume "pvc-adea4cc5-58b9-4e2d-b604-dae1222cd314" Volume is already used by pod(s) rook-ceph-osd-0-6d6b4ddf96-sh5qn Warning FailedMount 13m kubelet, ip-10-0-142-177.us-west-2.compute.internal Unable to attach or mount volumes: unmounted volumes=[set1-0-data-l8dpp], unattached volumes=[rook-ceph-osd-token-vfkgn run-udev set1-0-data-l8dpp-bridge rook-ceph-crash devices rook-config-override set1-0-data-l8dpp rook-ceph-log rook-binaries rook-data]: timed out waiting for the condition Warning FailedMount 11m kubelet, ip-10-0-142-177.us-west-2.compute.internal Unable to attach or mount volumes: unmounted volumes=[set1-0-data-l8dpp], unattached volumes=[set1-0-data-l8dpp-bridge run-udev rook-data rook-config-override rook-ceph-log rook-binaries rook-ceph-crash rook-ceph-osd-token-vfkgn devices set1-0-data-l8dpp]: timed out waiting for the condition Warning FailedMount 8m48s kubelet, ip-10-0-142-177.us-west-2.compute.internal Unable to attach or mount volumes: unmounted volumes=[set1-0-data-l8dpp], unattached volumes=[rook-binaries set1-0-data-l8dpp run-udev devices rook-data rook-config-override rook-ceph-log rook-ceph-crash rook-ceph-osd-token-vfkgn set1-0-data-l8dpp-bridge]: timed out waiting for the condition Warning FailedMount 6m33s kubelet, ip-10-0-142-177.us-west-2.compute.internal Unable to attach or mount volumes: unmounted volumes=[set1-0-data-l8dpp], unattached volumes=[rook-config-override rook-ceph-crash rook-binaries set1-0-data-l8dpp set1-0-data-l8dpp-bridge rook-data rook-ceph-log rook-ceph-osd-token-vfkgn devices run-udev]: timed out waiting for the condition Warning FailedMount 4m18s kubelet, ip-10-0-142-177.us-west-2.compute.internal Unable to attach or mount volumes: unmounted volumes=[set1-0-data-l8dpp], unattached volumes=[set1-0-data-l8dpp rook-binaries devices run-udev rook-config-override rook-ceph-crash rook-ceph-osd-token-vfkgn rook-data rook-ceph-log set1-0-data-l8dpp-bridge]: timed out waiting for the condition Warning FailedMount 2m2s kubelet, ip-10-0-142-177.us-west-2.compute.internal Unable to attach or mount volumes: unmounted volumes=[set1-0-data-l8dpp], unattached volumes=[rook-ceph-crash rook-data rook-ceph-osd-token-vfkgn set1-0-data-l8dpp set1-0-data-l8dpp-bridge rook-config-override devices rook-ceph-log run-udev rook-binaries]: timed out waiting for the condition So with the old OS now gone, the new OSD pod is still in the same state - the reason is that in EC2, when you shutdown a node, the gp2 volume is still attached to the downed node. from the AWS docs: "When you stop a running instance, the following happens: - The instance performs a normal shutdown and stops running; its status changes to stopping and then stopped. - Any Amazon EBS volumes remain attached to the instance, and their data persists." will update with 2nd test of removing a node completely from aws.
2nd test was done on same cluster configuration. I've accessed the AWS console and terminated one of the instances that had OSD pod on it. it took about 35 seconds from when pressing "terminate" until OCP marked the node down. the gp2 volume was free to move and in "available" state. same status where old osd is at "terminating" state and new osd at "Init 0/4". about 2 minutes in, I guess AWS sends a "good" signal to OCP, so the node is now gone from the cluster, which means the failed OSD is not showing up and the new OSD "picked up" the available gp2 volume and started. I think in this scenario, from viewing the logs, the new code was not even involved.
The upstream PR is merged, but before we backport to 4.4 we should confirm that it actually makes a difference in the IBM scenario. As Sagy showed, it doesn't help with AWS failure scenarios.
https://github.com/openshift/rook/pull/46 is merged to downstream rook release-4.4. Let's get a new RC build out for further testing for IBM.
Not sure if this is surprising or not but every application pod that will run with RWO, will have the same issue on node shutdown. Name: pod-test-rbd-1dc1c0d372fa41308c75c20d009c5795-1-m827z Namespace: namespace-test-0d6db3b374784e31a5bb352f52acd82c Priority: 0 PriorityClassName: <none> Node: ip-10-0-138-152.us-east-2.compute.internal/10.0.138.152 Start Time: Sun, 03 May 2020 12:37:40 +0300 Labels: deployment=pod-test-rbd-1dc1c0d372fa41308c75c20d009c5795-1 deploymentconfig=pod-test-rbd-1dc1c0d372fa41308c75c20d009c5795 name=pod-test-rbd-1dc1c0d372fa41308c75c20d009c5795 Annotations: openshift.io/deployment-config.latest-version: 1 openshift.io/deployment-config.name: pod-test-rbd-1dc1c0d372fa41308c75c20d009c5795 openshift.io/deployment.name: pod-test-rbd-1dc1c0d372fa41308c75c20d009c5795-1 openshift.io/scc: privileged Status: Pending IP: Controlled By: ReplicationController/pod-test-rbd-1dc1c0d372fa41308c75c20d009c5795-1 Containers: fedora: Container ID: Image: fedora Image ID: Port: <none> Host Port: <none> Command: /bin/bash -ce tail -f /dev/null State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Limits: cpu: 150m memory: 800Mi Requests: cpu: 150m memory: 800Mi Liveness: exec [sh -ec df /mnt] delay=3s timeout=1s period=3s #success=1 #failure=3 Environment: <none> Mounts: /mnt from fedora-vol (rw) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: fedora-vol: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc-test-1616b5f24bef4d059a3c99ece28662e8 ReadOnly: false QoS Class: Guaranteed Node-Selectors: dc=fedora Tolerations: node.kubernetes.io/memory-pressure:NoSchedule node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned namespace-test-0d6db3b374784e31a5bb352f52acd82c/pod-test-rbd-1dc1c0d372fa41308c75c20d009c5795-1-m827z to ip-10-0-138-152.us-east-2.compute.internal Warning FailedAttachVolume 67m attachdetach-controller Multi-Attach error for volume "pvc-d95a4433-4455-48cf-ad37-7fc3729c60df" Volume is already used by pod(s) pod-test-rbd-1dc1c0d372fa41308c75c20d009c5795-1-29845 Warning FailedMount 115s (x29 over 65m) kubelet, ip-10-0-138-152.us-east-2.compute.internal Unable to attach or mount volumes: unmounted volumes=[fedora-vol], unattached volumes=[fedora-vol]: timed out waiting for the condition (.venv) [~/git/ocs-ci]$ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-test-1616b5f24bef4d059a3c99ece28662e8 Bound pvc-d95a4433-4455-48cf-ad37-7fc3729c60df 3Gi RWO ocs-storagecluster-ceph-rbd 88m
This was tested on 4.3 and reproduced - not a regression
Tested with OCS 4.4 rc3 deployed on VMware (OCP 4.3.12). Results are not expected for force deleting the Terminating osd-0 pod. It did not recreate on a new OCP node for portable=true until I login to Ceph and do 'ceph osd out osd.0'. After marking osd.0 as 'out' the Terminating pod is gone (force delete) and the new osd-0 pod is created on compute-0 in ~10-12 mins. The Terminating MON pod (mon-b) does not get force deleted and does not recreate on new node. Also, to get new osd-0 to join the ceph cluster I had to again login to Ceph and do 'ceph osd in osd.0'. $ oc get nodes NAME STATUS ROLES AGE VERSION compute-0 Ready worker 113m v1.16.2 compute-1 NotReady worker 116m v1.16.2 compute-2 Ready worker 116m v1.16.2 compute-3 Ready worker 117m v1.16.2 compute-4 Ready worker 115m v1.16.2 compute-5 Ready worker 117m v1.16.2 control-plane-0 Ready master 117m v1.16.2 control-plane-1 Ready master 116m v1.16.2 control-plane-2 Ready master 117m v1.16.2 $ oc get csv NAME DISPLAY VERSION REPLACES PHASE ocs-operator.v4.4.0-416.ci OpenShift Container Storage 4.4.0-416.ci Succeeded $ oc get pods -o wide | grep osd | grep -v prepare rook-ceph-osd-0-dc6876c4f-26nkl 1/1 Terminating 0 69m 10.128.2.13 compute-1 <none> <none> rook-ceph-osd-0-dc6876c4f-hcmqv 0/1 Init:0/4 0 31m <none> compute-0 <none> <none> rook-ceph-osd-1-59bf6ccf9-ttg7t 1/1 Running 0 69m 10.130.0.14 compute-3 <none> <none> rook-ceph-osd-2-74968f887f-ngz2d 1/1 Running 0 68m 10.129.2.15 compute-2 <none> $ oc get pods -o wide | grep mon rook-ceph-mon-a-59b8b9c66c-mtpnq 1/1 Running 0 73m 10.130.0.11 compute-3 <none> <none> rook-ceph-mon-b-6b798bb594-j8tsr 0/1 Init:0/2 0 32m <none> compute-5 <none> <none> rook-ceph-mon-b-6b798bb594-t66wx 1/1 Terminating 0 72m 10.128.2.10 compute-1 <none> <none> rook-ceph-mon-c-f8485b559-22946 1/1 Running 0 72m 10.129.2.12 compute-2 <none> <none> rook-ceph-mon-d-5b55dc6f5b-w6ggr 1/1 Running 0 71m 10.130.2.9 compute-4 <none> <none> rook-ceph-mon-e-6c85c85dff-mjmxs 1/1 Running 0 71m 10.128.4.11 compute-0 <none> <none> rook-ceph-mon-f-canary-7bd8dcf55f-dcldq 0/1 Pending 0 22s <none> <none> <none> <none>
With the merged fix, the assumption was that ceph would automatically mark the OSD as out. But as we have seen, in certain configurations, Ceph does not mark the OSDs as out. In particular, when there are three nodes with one OSD per node, the OSD will not be marked out since there is no other OSD where the PGs can be moved. There really isn't a reason for Rook to wait for an OSD to be out before force deleting the pod. The other checks for the pod in the terminating state and the node being not ready are sufficient. I'd recommend we make this change as well for 4.4 if there is still time.
Moving back to Post with the fix as mentioned in the previous comment so we can still force delete the osd pod even if not marked as out by ceph. https://github.com/rook/rook/pull/5414
Per discussion in gchat, this doesn't block the 4.4 release. Here is the PR to merge the fix downstream if we think it should be included in 4.4.z. https://github.com/openshift/rook/pull/49
We had another blocker (see https://bugzilla.redhat.com/show_bug.cgi?id=1833082), so going ahead with this merge as well.
DTested RC5 behavior Deployed OCS 4.4 RC5 with the following 3 masters (m5.2xlarge) 6 workers (i3.4xlarge) Deployed OCS on 3 nodes Failed one node from AWS console (running mon.c, osd.0 and rook-operator) mon.c remains in terminating (old pod) + mon.c in init 0/4 (new pod) osd.0 remains in terminating (old pod) + osd.0 in init 0/4 (new pod) rokk-operator remains in terminating (old pod) + rook-operator in Running(new pod) Both new pods show the multi-attach error. After 45 minutes osd.0 terminating pod was effectively deleted by rook-ceph-operator New rook-operator pod start of log file 2020-05-11 17:59:29.727976 I | rookcmd: starting Rook 4.4-13.3a9bf923.release_4.4 with arguments '/usr/local/bin/rook ceph operator' 2020-05-11 17:59:29.728101 I | rookcmd: flag values: --add_dir_header=false, --alsologtostderr=false, --csi-attacher-image=registry.redhat.io/openshift4/ose-csi-external-attacher@sha256:fb9f73ed22b4241eba25e71b63aa6729da New rook-operator pod message when finally detecting the stuck osd.0 2020-05-11 18:45:19.962313 I | op-osd: force deleting pod "rook-ceph-osd-0-54c6c7ff5b-t5bht" that appears to be stuck terminating 2020-05-11 18:45:19.976498 I | op-osd: pod "rook-ceph-osd-0-54c6c7ff5b-t5bht" deletion succeeded From that point osd.0 goes to running state and the cluster recoevrs but still signals mon.c is DOWN. Manual delete of the terminating mon.c pod finally fixed the cluster. From Travis analysis the operator got stuck for 20 minutes waiting for the failed MON (mon.c) to respond, another 20 minutes for the failed OSD to respond (osd.0) and then the expected 5 minute timer to take action. Restarting the failed nodes had not effect and the cluster remained healthy. Attaching operator log to the ticket (oper2.txt)
Created attachment 1687459 [details] rook-operator log during failure. OCP 4.3.18 OCS 4.4.0 RC5
Test failing a node that only runs OSDs. 14:34:32 - Failed the AWS instance 14:48:xx - osd.4 old pod deleted and new pod in running state with cluster health All without me touching the keyboard. Attaching log file oper3.txt
Created attachment 1687476 [details] rook-ceph operator log when failing a node running only OSDs Same environment as previous test but failing a node running ony OSDs This all testing had protable=true and gp2 based OSDs (2TB)
Tested on AWS_IPI with 6 ocs nodes 3 master : m4.xlarge 6 ocs : m5.4xlarge Versions: --------- $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.0-0.nightly-2020-05-15-010811 True False 34m Cluster version is 4.4.0-0.nightly-2020-05-15-010811 $ oc get csv -n openshift-storage NAME DISPLAY VERSION REPLACES PHASE ocs-operator.v4.4.0-420.ci OpenShift Container Storage 4.4.0-420.ci Succeeded Steps perfomed: --------------- 1. Created 3 OCS node cluster. 2. Added 3 extra nodes and labelled nodes. $ oc label nodes ip-10-0-139-209.us-east-2.compute.internal ip-10-0-158-247.us-east-2.compute.internal ip-10-0-167-137.us-east-2.compute.internal cluster.ocs.openshift.io/openshift-storage= node/ip-10-0-139-209.us-east-2.compute.internal labeled node/ip-10-0-158-247.us-east-2.compute.internal labeled node/ip-10-0-167-137.us-east-2.compute.internal labeled 3. Power off node ip-10-0-145-21.us-east-2.compute.internal Note: - New node available on same zone ip-10-0-158-247.us-east-2.compute.internal - Pods existed on node ip-10-0-145-21.us-east-2.compute.internal $ oc get pods -n openshift-storage -o wide | grep ip-10-0-145-21.us-east-2.compute.internal csi-cephfsplugin-gk4s4 3/3 Running 0 36m 10.0.145.21 ip-10-0-145-21.us-east-2.compute.internal <none> <none> csi-cephfsplugin-provisioner-5d94649b9f-4pc7f 5/5 Running 0 36m 10.131.0.25 ip-10-0-145-21.us-east-2.compute.internal <none> <none> csi-rbdplugin-9dz6p 3/3 Running 0 36m 10.0.145.21 ip-10-0-145-21.us-east-2.compute.internal <none> <none> noobaa-core-0 1/1 Running 0 32m 10.131.0.35 ip-10-0-145-21.us-east-2.compute.internal <none> <none> noobaa-db-0 1/1 Running 0 32m 10.131.0.37 ip-10-0-145-21.us-east-2.compute.internal <none> <none> rook-ceph-crashcollector-ip-10-0-145-21-776dc5f8b6-92wpz 1/1 Running 0 35m 10.131.0.28 ip-10-0-145-21.us-east-2.compute.internal <none> <none> rook-ceph-drain-canary-7732c9c352d5ea4b10304ba350fad0a0-68wtt84 1/1 Running 0 32m 10.131.0.34 ip-10-0-145-21.us-east-2.compute.internal <none> <none> rook-ceph-mon-a-5c7bd899f-mj2ft 1/1 Running 0 35m 10.131.0.29 ip-10-0-145-21.us-east-2.compute.internal <none> <none> rook-ceph-operator-5f858bfb9f-6qmxz 1/1 Running 0 37m 10.131.0.24 ip-10-0-145-21.us-east-2.compute.internal <none> <none> rook-ceph-osd-2-545c6d4896-w88js 1/1 Running 0 32m 10.131.0.36 ip-10-0-145-21.us-east-2.compute.internal <none> <none> rook-ceph-osd-prepare-ocs-deviceset-2-0-dfkld-8thn5 0/1 Completed 0 32m 10.131.0.32 ip-10-0-145-21.us-east-2.compute.internal <none> <none> rook-ceph-tools-869df698dc-wpxd7 1/1 Running 0 32m 10.0.145.21 ip-10-0-145-21.us-east-2.compute.internal <none> <none> Observation: ------------ After ~45 minutes osd.2 terminating pod was forcefully deleted by rook-ceph-operator and later after ~7 minutes the new osd.2 pod came into running state. from rook-ceph-operator log: 2020-05-15 06:27:13.346650 I | op-osd: force deleting pod "rook-ceph-osd-2-545c6d4896-w88js" that appears to be stuck terminating 2020-05-15 06:27:13.364624 I | op-osd: pod "rook-ceph-osd-2-545c6d4896-w88js" deletion succeeded new osd.2 pod came into running state: Fri May 15 06:34:17 UTC 2020 ===PODS=== NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES rook-ceph-osd-2-545c6d4896-qr6gw 0/1 Init:3/4 0 53m 10.128.4.8 ip-10-0-158-247.us-east-2.compute.internal <none> <none> ++++++++++++++++++++ Fri May 15 06:34:25 UTC 2020 ===PODS=== NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES rook-ceph-osd-2-545c6d4896-qr6gw 1/1 Running 0 53m 10.128.4.8 ip-10-0-158-247.us-east-2.compute.internal <none> <none> But still old mon.a, rook-ceph-operator, nooba pods are in terminating state as shown below: $ oc get pods -n openshift-storage -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES noobaa-core-0 1/1 Terminating 0 4h4m 10.131.0.35 ip-10-0-145-21.us-east-2.compute.internal <none> <none> noobaa-db-0 1/1 Terminating 0 4h4m 10.131.0.37 ip-10-0-145-21.us-east-2.compute.internal <none> <none> noobaa-endpoint-9bc7977c9-55bmc 1/1 Running 26 4h3m 10.128.2.23 ip-10-0-169-205.us-east-2.compute.internal <none> <none> noobaa-operator-6c6c99b8b5-wvnjm 1/1 Running 0 4h9m 10.129.2.14 ip-10-0-129-136.us-east-2.compute.internal <none> <none> ocs-operator-544d9ddd9d-4s228 0/1 Running 0 4h9m 10.128.2.12 ip-10-0-169-205.us-east-2.compute.internal <none> <none> rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-cf9977b8qn8jl 1/1 Running 0 4h4m 10.128.2.22 ip-10-0-169-205.us-east-2.compute.internal <none> <none> rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-85dc8fc68jf57 1/1 Running 0 4h4m 10.129.2.24 ip-10-0-129-136.us-east-2.compute.internal <none> <none> rook-ceph-mgr-a-859d7b786b-drjtw 1/1 Running 0 4h6m 10.129.2.19 ip-10-0-129-136.us-east-2.compute.internal <none> <none> rook-ceph-mon-a-5c7bd899f-mj2ft 1/1 Terminating 0 4h7m 10.131.0.29 ip-10-0-145-21.us-east-2.compute.internal <none> <none> rook-ceph-mon-b-866cf745bb-lzknx 1/1 Running 0 4h6m 10.129.2.18 ip-10-0-129-136.us-east-2.compute.internal <none> <none> rook-ceph-mon-c-85ccc5d4cf-w5ctz 1/1 Running 0 4h6m 10.128.2.17 ip-10-0-169-205.us-east-2.compute.internal <none> <none> rook-ceph-mon-d-5f49d654c8-kqh46 1/1 Running 0 3h43m 10.130.2.6 ip-10-0-139-209.us-east-2.compute.internal <none> <none> rook-ceph-mon-e-95dc9df-7slgt 1/1 Running 0 3h42m 10.128.4.7 ip-10-0-158-247.us-east-2.compute.internal <none> <none> rook-ceph-mon-f-7f7d4b748-pj8zr 1/1 Running 0 119m 10.131.2.19 ip-10-0-167-137.us-east-2.compute.internal <none> <none> rook-ceph-operator-5f858bfb9f-6qmxz 1/1 Terminating 0 4h9m 10.131.0.24 ip-10-0-145-21.us-east-2.compute.internal <none> <none> rook-ceph-operator-5f858bfb9f-qqcjq 1/1 Running 0 171m 10.130.2.8 ip-10-0-139-209.us-east-2.compute.internal <none> <none> rook-ceph-osd-0-6fbd59c7d7-xrjfv 1/1 Running 0 4h4m 10.128.2.21 ip-10-0-169-205.us-east-2.compute.internal <none> <none> rook-ceph-osd-1-78499446cf-bshnb 1/1 Running 0 4h4m 10.129.2.23 ip-10-0-129-136.us-east-2.compute.internal <none> <none> rook-ceph-osd-2-545c6d4896-qr6gw 1/1 Running 0 171m 10.128.4.8 ip-10-0-158-247.us-east-2.compute.internal <none> <none> rook-ceph-osd-prepare-ocs-deviceset-0-0-2xzcc-nqjw9 0/1 Completed 0 4h4m 10.128.2.19 ip-10-0-169-205.us-east-2.compute.internal <none> <none> rook-ceph-osd-prepare-ocs-deviceset-1-0-lvpfg-2w4d7 0/1 Completed 0 4h4m 10.129.2.21 ip-10-0-129-136.us-east-2.compute.internal <none> <none> rook-ceph-tools-869df698dc-hd2gb 1/1 Running 0 171m 10.0.139.209 ip-10-0-139-209.us-east-2.compute.internal <none> <none> rook-ceph-tools-869df698dc-wpxd7 1/1 Terminating 0 4h4m 10.0.145.21 ip-10-0-145-21.us-east-2.compute.internal <none> <none>
Attached log: http://rhsqe-repo.lab.eng.blr.redhat.com/ocs4qe/akrai/bz1830015/power-off-node/rook-ceph-op/ Additional information can be found in bug1835908#c7
Tested on AWS IPI environment with different scenarios as following. Observed that in all cases osd pods came up and running. Version: OCP: 4.4.0-0.nightly-2020-05-22-000345 OCS: 4.4.0-428.ci Logs: http://rhsqe-repo.lab.eng.blr.redhat.com/ocs4qe/akrai/bz1830015/ Case 1: Drain node hosted with rook-operator 3 M, 3 W(OCS node) Drain node: ip-10-0-145-43.us-east-2.compute.internal Pods hosted on drain node: - noobaa-operator - rook-ceph-mds-ocs-storagecluster-cephfilesystem-b - rook-ceph-mgr-a - rook-ceph-mon-b - rook-ceph-operator - rook-ceph-osd-1 node drain at utc: Fri May 22 06:16:48 UTC 2020 Operator, mgr, mds pod was up and running on the other available ocs node. mon-b and osd-1 pod was in pending state as expected. node scheduled again at utc: Fri May 22 06:50:15 UTC 2020 Once the node is in ready state, mon-b and osd-1 pod up and running at utc Fri May 22 06:51:02 UTC 2020 Case 2: Drain node without rook-operator hosted on it 3 M, 6 W(OCS node) Drain node: ip-10-0-130-2.us-east-2.compute.internal Available node on same zone: ip-10-0-135-128.us-east-2.compute.internal Pods hosted on node: - noobaa-endpoint - rook-ceph-mds-ocs-storagecluster-cephfilesystem-a - rook-ceph-mgr-a - rook-ceph-mon-c - rook-ceph-osd-2 node drain at utc: Fri May 22 07:55:40 UTC 2020 Initially osd-2 pod was stuck in init 0/4 at utc Fri May 22 07:56:18 UTC 2020, with multi-attach error $ date --utc;oc describe pod rook-ceph-osd-2-56cc4567dd-sfkbr -n openshift-storage Fri May 22 07:58:08 UTC 2020 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 113s default-scheduler Successfully assigned openshift-storage/rook-ceph-osd-2-56cc4567dd-sfkbr to ip-10-0-135-128.us-east-2.compute.internal Warning FailedAttachVolume 113s attachdetach-controller Multi-Attach error for volume "pvc-ef9ddce4-1691-432d-96f9-9936127817ef" Volume is already used by pod(s) rook-ceph-osd-2-56cc4567dd-frx97 osd-2 pod came up and running state on available node at utc: Fri May 22 08:04:40 UTC 2020 node scheduled again at utc: Fri May 22 08:28:50 UTC 2020 Case 3: Shut down node without rook-operator hosted on it 3 M, 6 W(OCS node) Shut down node: ip-10-0-145-43.us-east-2.compute.internal Available node on same zone: ip-10-0-154-227.us-east-2.compute.internal Pods hosted on node: - rook-ceph-mon-b - rook-ceph-osd-1 Node shutdown at utc: Fri May 22 10:26:49 UTC 2020 Node reported as NotReady at utc: Fri May 22 10:27:17 UTC 2020 mon-b and osd-1 pod went to terminating state at utc Fri May 22 10:32:25 UTC 2020. New osd-1 pod created on available node at utc Fri May 22 10:32:25 UTC 2020 and stuck in Init:0/4 with multi-attach error. Fri May 22 10:32:25 UTC 2020 rook-ceph-osd-1-77cc8cd74b-dfrsk 1/1 Terminating 0 4h15m 10.129.2.39 ip-10-0-145-43.us-east-2.compute.internal <none> <none> rook-ceph-osd-1-77cc8cd74b-m59kg 0/1 Init:0/4 0 16s <none> ip-10-0-154-227.us-east-2.compute.internal <none> <none> old osd-1 pod was forcefully deleted at utc 2020-05-22 10:32:49 2020-05-22 10:32:49.165552 I | op-osd: force deleting pod "rook-ceph-osd-1-77cc8cd74b-dfrsk" that appears to be stuck terminating 2020-05-22 10:32:49.182520 I | op-osd: pod "rook-ceph-osd-1-77cc8cd74b-dfrsk" deletion succeeded new osd-1 pod came up and running at utc Fri May 22 10:40:39 UTC 2020
Tested on AWS and force deletion takes ~10 mins to kick after a node goes down And if the operator + osd are co-located - it can take up to ~45 min (tracked by another bug1835908) Do we need test on another cluster ?
VMWare Environment 6 worker nodes 3 OSDs $ oc version Client Version: 4.3.18 Server Version: 4.4.3 Kubernetes Version: v1.17.1 $ oc get csv NAME DISPLAY VERSION REPLACES PHASE lib-bucket-provisioner.v1.0.0 lib-bucket-provisioner 1.0.0 Succeeded ocs-operator.v4.4.0-428.ci OpenShift Container Storage 4.4.0-428.ci Succeeded 12:25 Shutdown Node compute-3 (mon, mds, osd but not the rook-operator) $ oc get nodes NAME STATUS ROLES AGE VERSION compute-0 Ready worker 2d6h v1.17.1 compute-1 Ready worker 2d6h v1.17.1 compute-2 Ready worker 2d6h v1.17.1 compute-3 Ready worker 2d6h v1.17.1 compute-4 Ready worker 2d6h v1.17.1 compute-5 Ready worker 2d6h v1.17.1 control-plane-0 Ready master 2d6h v1.17.1 control-plane-1 Ready master 2d6h v1.17.1 control-plane-2 Ready master 2d6h v1.17.1 12:25 Node goes not ready $ oc get nodes NAME STATUS ROLES AGE VERSION compute-0 Ready worker 2d6h v1.17.1 compute-1 Ready worker 2d6h v1.17.1 compute-2 Ready worker 2d6h v1.17.1 compute-3 NotReady worker 2d6h v1.17.1 compute-4 Ready worker 2d6h v1.17.1 compute-5 Ready worker 2d6h v1.17.1 control-plane-0 Ready master 2d6h v1.17.1 control-plane-1 Ready master 2d6h v1.17.1 control-plane-2 Ready master 2d6h v1.17.1 $ oc get pods NAME READY STATUS RESTARTS AGE csi-cephfsplugin-26w76 3/3 Running 0 2d csi-cephfsplugin-4rqq7 3/3 Running 0 2d csi-cephfsplugin-7bhx5 3/3 Running 0 2d csi-cephfsplugin-crxsf 3/3 Running 0 2d csi-cephfsplugin-g8kg4 3/3 Running 0 2d csi-cephfsplugin-ll2h6 3/3 Running 0 2d csi-cephfsplugin-provisioner-6b6cf67567-4j5r4 5/5 Running 0 2d csi-cephfsplugin-provisioner-6b6cf67567-brjdc 5/5 Running 0 2d csi-rbdplugin-2k2sr 3/3 Running 0 2d csi-rbdplugin-4fhhp 3/3 Running 0 2d csi-rbdplugin-5gxc6 3/3 Running 0 2d csi-rbdplugin-dcngd 3/3 Running 0 2d csi-rbdplugin-provisioner-7f76d8c686-jj7tp 5/5 Running 0 2d csi-rbdplugin-provisioner-7f76d8c686-tp6kf 5/5 Running 0 2d csi-rbdplugin-rhxlf 3/3 Running 0 2d csi-rbdplugin-wnnjk 3/3 Running 0 2d lib-bucket-provisioner-5b9cb4f848-6ktkx 1/1 Running 0 2d noobaa-core-0 1/1 Running 0 2d noobaa-db-0 1/1 Running 0 2d noobaa-endpoint-769c6b8d4c-l99lg 1/1 Running 0 2d noobaa-operator-7f55c9477c-zbdnf 1/1 Running 0 2d ocs-operator-c59d97cc4-5tnsp 1/1 Running 0 2d rook-ceph-crashcollector-compute-0-55bd8f5b5f-dq495 1/1 Running 0 2d rook-ceph-crashcollector-compute-1-6857864dbd-wq4nv 1/1 Running 0 2d rook-ceph-crashcollector-compute-2-6ff5f55749-gq7zg 1/1 Running 0 2d rook-ceph-crashcollector-compute-3-77bc6bbc66-klnkx 1/1 Running 0 2d rook-ceph-crashcollector-compute-5-dcb77cbd-d86kc 1/1 Running 0 2d rook-ceph-drain-canary-compute-0-77f57f7b4f-h64xb 1/1 Running 0 2d rook-ceph-drain-canary-compute-1-66fbf47894-j8k8c 1/1 Running 0 2d rook-ceph-drain-canary-compute-3-d4cd9b9c6-5szs4 1/1 Running 0 44h rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-69d7646f56xgv 1/1 Running 0 2d rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-b858d6bfpd2fx 1/1 Running 0 2d rook-ceph-mgr-a-64f8854744-knsjw 1/1 Running 0 2d rook-ceph-mon-a-79c56c6945-grwl8 1/1 Running 0 2d rook-ceph-mon-b-9d4cc4c7d-5dh8b 1/1 Running 0 2d rook-ceph-mon-c-8585f79df-wk5ns 1/1 Running 0 2d rook-ceph-mon-d-5bf4864564-2q6ws 1/1 Running 0 2d rook-ceph-mon-e-76bc44495c-qjnk4 1/1 Running 0 2d rook-ceph-operator-544c7c6469-xkq22 1/1 Running 0 44h rook-ceph-osd-0-bf765b6b9-s2zsb 1/1 Running 0 44h rook-ceph-osd-1-558597f74f-xqlpl 1/1 Running 0 2d rook-ceph-osd-2-8d99fb959-gllxz 1/1 Running 0 2d rook-ceph-osd-prepare-ocs-deviceset-0-0-mvprc-ckcgw 0/1 Completed 0 44h rook-ceph-osd-prepare-ocs-deviceset-1-0-wjwsx-2mjkb 0/1 Completed 0 2d rook-ceph-osd-prepare-ocs-deviceset-2-0-9d5ms-t62fr 0/1 Completed 0 2d rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-b4f74b7lqx9r 1/1 Running 0 2d rook-ceph-tools-698cc696f9-q5pfr 1/1 Running 0 47h # ceph -s cluster: id: 203b1074-28d1-47da-bb00-0ba5cb02aa00 health: HEALTH_WARN 1 osds down 2 hosts (1 osds) down 1 rack (1 osds) down Degraded data redundancy: 409/1227 objects degraded (33.333%), 98 pgs degraded, 200 pgs undersized 1/5 mons down, quorum a,b,d,e services: mon: 5 daemons, quorum a,b,d,e (age 5m), out of quorum: c mgr: a(active, since 2d) mds: ocs-storagecluster-cephfilesystem:1 {0=ocs-storagecluster-cephfilesystem-a=up:active} 1 up:standby-replay osd: 3 osds: 2 up (since 5m), 3 in (since 45h) rgw: 1 daemon active (ocs.storagecluster.cephobjectstore.a) task status: data: pools: 11 pools, 200 pgs objects: 409 objects, 390 MiB usage: 4.0 GiB used, 1.5 TiB / 1.5 TiB avail pgs: 409/1227 objects degraded (33.333%) 102 active+undersized 98 active+undersized+degraded io: client: 1023 B/s rd, 8.2 KiB/s wr, 1 op/s rd, 1 op/s wr 12:30 Pod go into terminating state $ oc get pods NAME READY STATUS RESTARTS AGE csi-cephfsplugin-26w76 3/3 Running 0 2d csi-cephfsplugin-4rqq7 3/3 Running 0 2d csi-cephfsplugin-7bhx5 3/3 Running 0 2d csi-cephfsplugin-crxsf 3/3 Running 0 2d csi-cephfsplugin-g8kg4 3/3 Running 0 2d csi-cephfsplugin-ll2h6 3/3 Running 0 2d csi-cephfsplugin-provisioner-6b6cf67567-4j5r4 5/5 Terminating 0 2d csi-cephfsplugin-provisioner-6b6cf67567-brjdc 5/5 Running 0 2d csi-cephfsplugin-provisioner-6b6cf67567-zkx9h 0/5 ContainerCreating 0 34s csi-rbdplugin-2k2sr 3/3 Running 0 2d csi-rbdplugin-4fhhp 3/3 Running 0 2d csi-rbdplugin-5gxc6 3/3 Running 0 2d csi-rbdplugin-dcngd 3/3 Running 0 2d csi-rbdplugin-provisioner-7f76d8c686-jj7tp 5/5 Running 0 2d csi-rbdplugin-provisioner-7f76d8c686-tp6kf 5/5 Running 0 2d csi-rbdplugin-rhxlf 3/3 Running 0 2d csi-rbdplugin-wnnjk 3/3 Running 0 2d lib-bucket-provisioner-5b9cb4f848-6ktkx 1/1 Running 0 2d noobaa-core-0 1/1 Running 0 2d noobaa-db-0 1/1 Running 0 2d noobaa-endpoint-769c6b8d4c-l99lg 1/1 Running 0 2d noobaa-operator-7f55c9477c-zbdnf 1/1 Running 0 2d ocs-operator-c59d97cc4-5tnsp 1/1 Running 0 2d rook-ceph-crashcollector-compute-0-55bd8f5b5f-dq495 1/1 Running 0 2d rook-ceph-crashcollector-compute-1-6857864dbd-wq4nv 1/1 Running 0 2d rook-ceph-crashcollector-compute-2-6ff5f55749-gq7zg 1/1 Running 0 2d rook-ceph-crashcollector-compute-3-77bc6bbc66-klnkx 1/1 Terminating 0 2d rook-ceph-crashcollector-compute-3-77bc6bbc66-sm7c8 0/1 Pending 0 34s rook-ceph-crashcollector-compute-4-79cf9f8979-hl7j6 1/1 Running 0 34s rook-ceph-crashcollector-compute-5-dcb77cbd-d86kc 1/1 Running 0 2d rook-ceph-drain-canary-compute-0-77f57f7b4f-h64xb 1/1 Running 0 2d rook-ceph-drain-canary-compute-1-66fbf47894-j8k8c 1/1 Running 0 2d rook-ceph-drain-canary-compute-3-d4cd9b9c6-5szs4 1/1 Terminating 0 45h rook-ceph-drain-canary-compute-3-d4cd9b9c6-djp6q 0/1 Pending 0 34s rook-ceph-drain-canary-compute-4-57ffb59db4-r8bgt 0/1 ContainerCreating 0 34s rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-69d7646f56xgv 1/1 Running 0 2d rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-b858d6bfgf8hg 1/1 Running 0 34s rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-b858d6bfpd2fx 1/1 Terminating 0 2d rook-ceph-mgr-a-64f8854744-knsjw 1/1 Running 0 2d rook-ceph-mon-a-79c56c6945-grwl8 1/1 Running 0 2d rook-ceph-mon-b-9d4cc4c7d-5dh8b 1/1 Running 0 2d rook-ceph-mon-c-8585f79df-kqrk9 0/1 Init:0/2 0 34s rook-ceph-mon-c-8585f79df-wk5ns 1/1 Terminating 0 2d rook-ceph-mon-d-5bf4864564-2q6ws 1/1 Running 0 2d rook-ceph-mon-e-76bc44495c-qjnk4 1/1 Running 0 2d rook-ceph-operator-544c7c6469-xkq22 1/1 Running 0 45h rook-ceph-osd-0-bf765b6b9-ncm2w 0/1 Init:0/4 0 34s rook-ceph-osd-0-bf765b6b9-s2zsb 1/1 Terminating 0 45h rook-ceph-osd-1-558597f74f-xqlpl 1/1 Running 0 2d rook-ceph-osd-2-8d99fb959-gllxz 1/1 Running 0 2d rook-ceph-osd-prepare-ocs-deviceset-1-0-wjwsx-2mjkb 0/1 Completed 0 2d rook-ceph-osd-prepare-ocs-deviceset-2-0-9d5ms-t62fr 0/1 Completed 0 2d rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-b4f74b7lqx9r 1/1 Running 0 2d rook-ceph-tools-698cc696f9-q5pfr 1/1 Running 0 47h 12:31|32 OSD 0 automatically deleted by rook-operator $ oc get pods NAME READY STATUS RESTARTS AGE csi-cephfsplugin-26w76 3/3 Running 0 2d csi-cephfsplugin-4rqq7 3/3 Running 0 2d csi-cephfsplugin-7bhx5 3/3 Running 0 2d csi-cephfsplugin-crxsf 3/3 Running 0 2d csi-cephfsplugin-g8kg4 3/3 Running 0 2d csi-cephfsplugin-ll2h6 3/3 Running 0 2d csi-cephfsplugin-provisioner-6b6cf67567-4j5r4 5/5 Terminating 0 2d csi-cephfsplugin-provisioner-6b6cf67567-brjdc 5/5 Running 0 2d csi-cephfsplugin-provisioner-6b6cf67567-zkx9h 5/5 Running 0 102s csi-rbdplugin-2k2sr 3/3 Running 0 2d csi-rbdplugin-4fhhp 3/3 Running 0 2d csi-rbdplugin-5gxc6 3/3 Running 0 2d csi-rbdplugin-dcngd 3/3 Running 0 2d csi-rbdplugin-provisioner-7f76d8c686-jj7tp 5/5 Running 0 2d csi-rbdplugin-provisioner-7f76d8c686-tp6kf 5/5 Running 0 2d csi-rbdplugin-rhxlf 3/3 Running 0 2d csi-rbdplugin-wnnjk 3/3 Running 0 2d lib-bucket-provisioner-5b9cb4f848-6ktkx 1/1 Running 0 2d noobaa-core-0 1/1 Running 0 2d noobaa-db-0 1/1 Running 0 2d noobaa-endpoint-769c6b8d4c-l99lg 1/1 Running 0 2d noobaa-operator-7f55c9477c-zbdnf 1/1 Running 0 2d ocs-operator-c59d97cc4-5tnsp 1/1 Running 0 2d rook-ceph-crashcollector-compute-0-55bd8f5b5f-dq495 1/1 Running 0 2d rook-ceph-crashcollector-compute-1-6857864dbd-wq4nv 1/1 Running 0 2d rook-ceph-crashcollector-compute-2-6ff5f55749-gq7zg 1/1 Running 0 2d rook-ceph-crashcollector-compute-3-77bc6bbc66-klnkx 1/1 Terminating 0 2d rook-ceph-crashcollector-compute-3-77bc6bbc66-sm7c8 0/1 Pending 0 102s rook-ceph-crashcollector-compute-4-79cf9f8979-hl7j6 1/1 Running 0 102s rook-ceph-crashcollector-compute-5-dcb77cbd-d86kc 1/1 Running 0 2d rook-ceph-drain-canary-compute-0-77f57f7b4f-h64xb 1/1 Running 0 2d rook-ceph-drain-canary-compute-1-66fbf47894-j8k8c 1/1 Running 0 2d rook-ceph-drain-canary-compute-3-d4cd9b9c6-5szs4 1/1 Terminating 0 45h rook-ceph-drain-canary-compute-3-d4cd9b9c6-djp6q 0/1 Pending 0 102s rook-ceph-drain-canary-compute-4-57ffb59db4-r8bgt 1/1 Running 0 102s rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-69d7646f56xgv 1/1 Running 0 2d rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-b858d6bfgf8hg 1/1 Running 0 102s rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-b858d6bfpd2fx 1/1 Terminating 0 2d rook-ceph-mgr-a-64f8854744-knsjw 1/1 Running 0 2d rook-ceph-mon-a-79c56c6945-grwl8 1/1 Running 0 2d rook-ceph-mon-b-9d4cc4c7d-5dh8b 1/1 Running 0 2d rook-ceph-mon-c-8585f79df-kqrk9 0/1 Init:0/2 0 102s rook-ceph-mon-c-8585f79df-wk5ns 1/1 Terminating 0 2d rook-ceph-mon-d-5bf4864564-2q6ws 1/1 Running 0 2d rook-ceph-mon-e-76bc44495c-qjnk4 1/1 Running 0 2d rook-ceph-operator-544c7c6469-xkq22 1/1 Running 0 45h rook-ceph-osd-0-bf765b6b9-ncm2w 0/1 Init:0/4 0 102s rook-ceph-osd-1-558597f74f-xqlpl 1/1 Running 0 2d rook-ceph-osd-2-8d99fb959-gllxz 1/1 Running 0 2d rook-ceph-osd-prepare-ocs-deviceset-1-0-wjwsx-2mjkb 0/1 Completed 0 2d rook-ceph-osd-prepare-ocs-deviceset-2-0-9d5ms-t62fr 0/1 Completed 0 2d rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-b4f74b7lqx9r 1/1 Running 0 2d rook-ceph-tools-698cc696f9-q5pfr 1/1 Running 0 47h 09:33 Terminating mon pod on node $ oc delete pod rook-ceph-mon-c-8585f79df-wk5ns --force --grace-period=0 09:33 Terminating mds pod on node $ oc delete pod rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-b858d6bfpd2fx --force --grace-period=0 09:39 All OSD pods up and running $ oc get pods NAME READY STATUS RESTARTS AGE csi-cephfsplugin-26w76 3/3 Running 0 2d csi-cephfsplugin-4rqq7 3/3 Running 0 2d csi-cephfsplugin-7bhx5 3/3 Running 0 2d csi-cephfsplugin-crxsf 3/3 Running 0 2d csi-cephfsplugin-g8kg4 3/3 Running 0 2d csi-cephfsplugin-ll2h6 3/3 Running 0 2d csi-cephfsplugin-provisioner-6b6cf67567-4j5r4 5/5 Terminating 0 2d csi-cephfsplugin-provisioner-6b6cf67567-brjdc 5/5 Running 0 2d csi-cephfsplugin-provisioner-6b6cf67567-zkx9h 5/5 Running 0 9m20s csi-rbdplugin-2k2sr 3/3 Running 0 2d csi-rbdplugin-4fhhp 3/3 Running 0 2d csi-rbdplugin-5gxc6 3/3 Running 0 2d csi-rbdplugin-dcngd 3/3 Running 0 2d csi-rbdplugin-provisioner-7f76d8c686-jj7tp 5/5 Running 0 2d csi-rbdplugin-provisioner-7f76d8c686-tp6kf 5/5 Running 0 2d csi-rbdplugin-rhxlf 3/3 Running 0 2d csi-rbdplugin-wnnjk 3/3 Running 0 2d lib-bucket-provisioner-5b9cb4f848-6ktkx 1/1 Running 0 2d noobaa-core-0 1/1 Running 0 2d noobaa-db-0 1/1 Running 0 2d noobaa-endpoint-769c6b8d4c-l99lg 1/1 Running 0 2d noobaa-operator-7f55c9477c-zbdnf 1/1 Running 0 2d ocs-operator-c59d97cc4-5tnsp 1/1 Running 0 2d rook-ceph-crashcollector-compute-0-55bd8f5b5f-dq495 1/1 Running 0 2d rook-ceph-crashcollector-compute-1-6857864dbd-wq4nv 1/1 Running 0 2d rook-ceph-crashcollector-compute-2-6ff5f55749-gq7zg 1/1 Running 0 2d rook-ceph-crashcollector-compute-3-77bc6bbc66-klnkx 1/1 Terminating 0 2d rook-ceph-crashcollector-compute-3-77bc6bbc66-sm7c8 0/1 Pending 0 9m20s rook-ceph-crashcollector-compute-4-79cf9f8979-hl7j6 1/1 Running 0 9m20s rook-ceph-crashcollector-compute-5-dcb77cbd-d86kc 1/1 Running 0 2d rook-ceph-drain-canary-compute-0-77f57f7b4f-h64xb 1/1 Running 0 2d rook-ceph-drain-canary-compute-1-66fbf47894-j8k8c 1/1 Running 0 2d rook-ceph-drain-canary-compute-3-d4cd9b9c6-5szs4 1/1 Terminating 0 45h rook-ceph-drain-canary-compute-3-d4cd9b9c6-djp6q 0/1 Pending 0 9m20s rook-ceph-drain-canary-compute-4-57ffb59db4-r8bgt 1/1 Running 0 9m20s rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-69d7646f56xgv 1/1 Running 0 2d rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-b858d6bfgf8hg 1/1 Running 0 9m20s rook-ceph-mgr-a-64f8854744-knsjw 1/1 Running 0 2d rook-ceph-mon-a-79c56c6945-grwl8 1/1 Running 0 2d rook-ceph-mon-b-9d4cc4c7d-5dh8b 1/1 Running 0 2d rook-ceph-mon-c-8585f79df-kqrk9 0/1 Init:0/2 0 9m20s rook-ceph-mon-d-5bf4864564-2q6ws 1/1 Running 0 2d rook-ceph-mon-e-76bc44495c-qjnk4 1/1 Running 0 2d rook-ceph-mon-f-canary-5cc7c8d9bf-srfz9 0/1 Pending 0 36s rook-ceph-operator-544c7c6469-xkq22 1/1 Running 0 45h rook-ceph-osd-0-bf765b6b9-ncm2w 1/1 Running 0 9m20s rook-ceph-osd-1-558597f74f-xqlpl 1/1 Running 0 2d rook-ceph-osd-2-8d99fb959-gllxz 1/1 Running 0 2d rook-ceph-osd-prepare-ocs-deviceset-1-0-wjwsx-2mjkb 0/1 Completed 0 2d rook-ceph-osd-prepare-ocs-deviceset-2-0-9d5ms-t62fr 0/1 Completed 0 2d rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-b4f74b7lqx9r 1/1 Running 0 2d rook-ceph-tools-698cc696f9-q5pfr 1/1 Running 0 2d It took a total of about 14 minutes to go through the entire cycle. Attaching rook-operator log 20200522-vmw-nodeoff-withosdonly.log
Created attachment 1691173 [details] Rook Operator log Rook Operator log during the test
Based on comment26, comment28 and comment30, force deletion of old osd pod kicked in and all pods where up and running. Apart from the ~45 mins issue when operator + osd together (which tracked by https://bugzilla.redhat.com/show_bug.cgi?id=1835908), didn't see any other issues, so moving this bz to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2393