Fedora Account System
Red Hat Associate
Red Hat Customer
Replacing failed node is it required to restart mgr pod Reported by: aclewett https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.3/html/managing_openshift_container_storage/replacing-storage-nodes-for-openshift-container-storage_rhocs#annotations:089792f3-215b-44f5-8ce1-25ca43bd59a1
Hi Erin, There's a good chance this is a workaround for another bug. @Elad, @Neha, You will know better, could you keep me honest here?
Hi Erin Does this BZ need to be verified from 4.4 or 4.5 docs or both ? Is there a separate BZ to track the changes in 4.5 ? Current 4.5 docs have 5 instances of "oc delete pod rook-ceph-mgr-xxxx". Same with 4.4 docs as well "https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.5/html-single/managing_openshift_container_storage/index?lb_target=preview#replacing-a-local-storage-device_rhocs https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.4/html-single/managing_openshift_container_storage/index?lb_target=preview#scaling_up_storage_by_adding_capacity_to_your_openshift_container_storage_nodes_on_aws_or_vmware_infrastructure
I can confirm that with AWS UPI, OCP 4.5, OCS 4.5 we don't need to restart the mgr pod after node replacement. Notice that the old node name is "ip-10-0-51-23.us-east-2.compute.internal", and new node name is "ip-10-0-48-18.us-east-2.compute.internal": Here are the results before the node replacement: (ocs-ci-venv) [yitzhak@localhost ocs-ci]$ oc rsh -n openshift-storage $(oc get pods -o wide -n openshift-storage|grep tool|awk '{print$1}') ceph osd status +----+-------------------------------------------+-------+-------+--------+---------+--------+---------+-----------+ | id | host | used | avail | wr ops | wr data | rd ops | rd data | state | +----+-------------------------------------------+-------+-------+--------+---------+--------+---------+-----------+ | 0 | ip-10-0-92-92.us-east-2.compute.internal | 1399M | 510G | 0 | 2457 | 2 | 106 | exists,up | | 1 | ip-10-0-51-23.us-east-2.compute.internal | 1400M | 510G | 5 | 35.2k | 0 | 0 | exists,up | | 2 | ip-10-0-76-201.us-east-2.compute.internal | 1399M | 510G | 6 | 93.6k | 0 | 0 | exists,up | (ocs-ci-venv) [yitzhak@localhost ocs-ci]$ oc rsh -n openshift-storage $(oc get pods -o wide -n openshift-storage|grep tool|awk '{print$1}') ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.50000 root default -5 1.50000 region us-east-2 -4 0.50000 zone us-east-2a -3 0.50000 host ocs-deviceset-1-data-0-8m827 1 ssd 0.50000 osd.1 up 1.00000 1.00000 -14 0.50000 zone us-east-2b -13 0.50000 host ocs-deviceset-2-data-0-dqwjd 2 ssd 0.50000 osd.2 up 1.00000 1.00000 -10 0.50000 zone us-east-2c -9 0.50000 host ocs-deviceset-0-data-0-9dtxx 0 ssd 0.50000 osd.0 up 1.00000 1.00000 (ocs-ci-venv) [yitzhak@localhost ocs-ci]$ oc get pods -n openshift-storage | grep osd rook-ceph-osd-0-6dcd66cfc8-9hdtk 1/1 Running 0 25m rook-ceph-osd-1-5756686468-59ngb 1/1 Running 0 25m rook-ceph-osd-2-86449bb986-srl6x 1/1 Running 0 25m rook-ceph-osd-prepare-ocs-deviceset-0-data-0-9dtxx-hwqkp 0/1 Completed 0 25m rook-ceph-osd-prepare-ocs-deviceset-1-data-0-8m827-brnqd 0/1 Completed 0 25m rook-ceph-osd-prepare-ocs-deviceset-2-data-0-dqwjd-wxkh6 0/1 Completed 0 25m And here are the results after node replacement(without restarting the mgr pod): (ocs-ci-venv) [yitzhak@localhost ocs-ci]$ oc rsh -n openshift-storage $(oc get pods -o wide -n openshift-storage|grep tool|awk '{print$1}') ceph osd status +----+-------------------------------------------+-------+-------+--------+---------+--------+---------+-----------+ | id | host | used | avail | wr ops | wr data | rd ops | rd data | state | +----+-------------------------------------------+-------+-------+--------+---------+--------+---------+-----------+ | 0 | ip-10-0-92-92.us-east-2.compute.internal | 1738M | 510G | 0 | 10.4k | 2 | 106 | exists,up | | 1 | ip-10-0-48-18.us-east-2.compute.internal | 1738M | 510G | 0 | 2457 | 0 | 0 | exists,up | | 2 | ip-10-0-76-201.us-east-2.compute.internal | 1738M | 510G | 0 | 3276 | 0 | 0 | exists,up | +----+-------------------------------------------+-------+-------+--------+---------+--------+---------+-----------+ (ocs-ci-venv) [yitzhak@localhost ocs-ci]$ oc rsh -n openshift-storage $(oc get pods -o wide -n openshift-storage|grep tool|awk '{print$1}') ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.50000 root default -5 1.50000 region us-east-2 -4 0.50000 zone us-east-2a -3 0.50000 host ocs-deviceset-1-data-0-8m827 1 ssd 0.50000 osd.1 up 1.00000 1.00000 -14 0.50000 zone us-east-2b -13 0.50000 host ocs-deviceset-2-data-0-dqwjd 2 ssd 0.50000 osd.2 up 1.00000 1.00000 -10 0.50000 zone us-east-2c -9 0.50000 host ocs-deviceset-0-data-0-9dtxx 0 ssd 0.50000 osd.0 u (ocs-ci-venv) [yitzhak@localhost ocs-ci]$ oc get pods -n openshift-storage | grep osd rook-ceph-osd-0-6dcd66cfc8-9hdtk 1/1 Running 0 63m rook-ceph-osd-1-5756686468-t25gr 1/1 Running 0 17m rook-ceph-osd-2-86449bb986-srl6x 1/1 Running 0 62m rook-ceph-osd-prepare-ocs-deviceset-0-data-0-9dtxx-hwqkp 0/1 Completed 0 63m rook-ceph-osd-prepare-ocs-deviceset-2-data-0-dqwjd-wxkh6 0/1 Completed 0 63m
Addintional information about the cluster I used: OCP version: Client Version: 4.3.8 Server Version: 4.5.0-0.nightly-2020-08-01-204100 Kubernetes Version: v1.18.3+08c38ef OCS verison: cluster version NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.0-0.nightly-2020-08-01-204100 True False 139m Cluster version is 4.5.0-0.nightly-2020-08-01-204100 Rook version Ceph version OCP version: Client Version: 4.3.8 Server Version: 4.5.0-0.nightly-2020-08-31-101523 Kubernetes Version: v1.18.3+6c42de8 OCS verison: ocs-operator.v4.5.0-67.ci OpenShift Container Storage 4.5.0-67.ci Succeeded cluster version NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.0-0.nightly-2020-08-31-101523 True False 3h14m Cluster version is 4.5.0-0.nightly-2020-08-31-101523 Rook version rook: 4.5-46.d1b1f29c.release_4.5 go: go1.13.4 Ceph version ceph version 14.2.8-91.el8cp (75b4845da7d469665bd48d1a49badcc3677bf5cd) nautilus (stable)
Created attachment 1713436 [details] node replacement logs AWS UPI