Bug 1830175
| Summary: | [DDF] Replacing failed node is it required to restart mgr pod | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Container Storage | Reporter: | Direct Docs Feedback <ddf-bot> | ||||
| Component: | documentation | Assignee: | Erin Donnelly <edonnell> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Itzhak <ikave> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 4.3 | CC: | asriram, ebenahar, edonnell, kramdoss, nberry, ocs-bugs, ratamir, sagrawal | ||||
| Target Milestone: | --- | ||||||
| Target Release: | OCS 4.5.0 | ||||||
| Hardware: | All | ||||||
| OS: | All | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | Type: | --- | |||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1826397 | ||||||
| Attachments: |
|
||||||
|
Description
Direct Docs Feedback
2020-05-01 01:06:41 UTC
Hi Erin, There's a good chance this is a workaround for another bug. @Elad, @Neha, You will know better, could you keep me honest here? Hi Erin Does this BZ need to be verified from 4.4 or 4.5 docs or both ? Is there a separate BZ to track the changes in 4.5 ? Current 4.5 docs have 5 instances of "oc delete pod rook-ceph-mgr-xxxx". Same with 4.4 docs as well "https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.5/html-single/managing_openshift_container_storage/index?lb_target=preview#replacing-a-local-storage-device_rhocs https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.4/html-single/managing_openshift_container_storage/index?lb_target=preview#scaling_up_storage_by_adding_capacity_to_your_openshift_container_storage_nodes_on_aws_or_vmware_infrastructure I can confirm that with AWS UPI, OCP 4.5, OCS 4.5 we don't need to restart the mgr pod after node replacement. Notice that the old node name is "ip-10-0-51-23.us-east-2.compute.internal", and new node name is "ip-10-0-48-18.us-east-2.compute.internal": Here are the results before the node replacement:
(ocs-ci-venv) [yitzhak@localhost ocs-ci]$ oc rsh -n openshift-storage $(oc get pods -o wide -n openshift-storage|grep tool|awk '{print$1}') ceph osd status
+----+-------------------------------------------+-------+-------+--------+---------+--------+---------+-----------+
| id | host | used | avail | wr ops | wr data | rd ops | rd data | state |
+----+-------------------------------------------+-------+-------+--------+---------+--------+---------+-----------+
| 0 | ip-10-0-92-92.us-east-2.compute.internal | 1399M | 510G | 0 | 2457 | 2 | 106 | exists,up |
| 1 | ip-10-0-51-23.us-east-2.compute.internal | 1400M | 510G | 5 | 35.2k | 0 | 0 | exists,up |
| 2 | ip-10-0-76-201.us-east-2.compute.internal | 1399M | 510G | 6 | 93.6k | 0 | 0 | exists,up |
(ocs-ci-venv) [yitzhak@localhost ocs-ci]$ oc rsh -n openshift-storage $(oc get pods -o wide -n openshift-storage|grep tool|awk '{print$1}') ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 1.50000 root default
-5 1.50000 region us-east-2
-4 0.50000 zone us-east-2a
-3 0.50000 host ocs-deviceset-1-data-0-8m827
1 ssd 0.50000 osd.1 up 1.00000 1.00000
-14 0.50000 zone us-east-2b
-13 0.50000 host ocs-deviceset-2-data-0-dqwjd
2 ssd 0.50000 osd.2 up 1.00000 1.00000
-10 0.50000 zone us-east-2c
-9 0.50000 host ocs-deviceset-0-data-0-9dtxx
0 ssd 0.50000 osd.0 up 1.00000 1.00000
(ocs-ci-venv) [yitzhak@localhost ocs-ci]$ oc get pods -n openshift-storage | grep osd
rook-ceph-osd-0-6dcd66cfc8-9hdtk 1/1 Running 0 25m
rook-ceph-osd-1-5756686468-59ngb 1/1 Running 0 25m
rook-ceph-osd-2-86449bb986-srl6x 1/1 Running 0 25m
rook-ceph-osd-prepare-ocs-deviceset-0-data-0-9dtxx-hwqkp 0/1 Completed 0 25m
rook-ceph-osd-prepare-ocs-deviceset-1-data-0-8m827-brnqd 0/1 Completed 0 25m
rook-ceph-osd-prepare-ocs-deviceset-2-data-0-dqwjd-wxkh6 0/1 Completed 0 25m
And here are the results after node replacement(without restarting the mgr pod):
(ocs-ci-venv) [yitzhak@localhost ocs-ci]$ oc rsh -n openshift-storage $(oc get pods -o wide -n openshift-storage|grep tool|awk '{print$1}') ceph osd status
+----+-------------------------------------------+-------+-------+--------+---------+--------+---------+-----------+
| id | host | used | avail | wr ops | wr data | rd ops | rd data | state |
+----+-------------------------------------------+-------+-------+--------+---------+--------+---------+-----------+
| 0 | ip-10-0-92-92.us-east-2.compute.internal | 1738M | 510G | 0 | 10.4k | 2 | 106 | exists,up |
| 1 | ip-10-0-48-18.us-east-2.compute.internal | 1738M | 510G | 0 | 2457 | 0 | 0 | exists,up |
| 2 | ip-10-0-76-201.us-east-2.compute.internal | 1738M | 510G | 0 | 3276 | 0 | 0 | exists,up |
+----+-------------------------------------------+-------+-------+--------+---------+--------+---------+-----------+
(ocs-ci-venv) [yitzhak@localhost ocs-ci]$ oc rsh -n openshift-storage $(oc get pods -o wide -n openshift-storage|grep tool|awk '{print$1}') ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 1.50000 root default
-5 1.50000 region us-east-2
-4 0.50000 zone us-east-2a
-3 0.50000 host ocs-deviceset-1-data-0-8m827
1 ssd 0.50000 osd.1 up 1.00000 1.00000
-14 0.50000 zone us-east-2b
-13 0.50000 host ocs-deviceset-2-data-0-dqwjd
2 ssd 0.50000 osd.2 up 1.00000 1.00000
-10 0.50000 zone us-east-2c
-9 0.50000 host ocs-deviceset-0-data-0-9dtxx
0 ssd 0.50000 osd.0 u
(ocs-ci-venv) [yitzhak@localhost ocs-ci]$ oc get pods -n openshift-storage | grep osd
rook-ceph-osd-0-6dcd66cfc8-9hdtk 1/1 Running 0 63m
rook-ceph-osd-1-5756686468-t25gr 1/1 Running 0 17m
rook-ceph-osd-2-86449bb986-srl6x 1/1 Running 0 62m
rook-ceph-osd-prepare-ocs-deviceset-0-data-0-9dtxx-hwqkp 0/1 Completed 0 63m
rook-ceph-osd-prepare-ocs-deviceset-2-data-0-dqwjd-wxkh6 0/1 Completed 0 63m
Addintional information about the cluster I used: OCP version: Client Version: 4.3.8 Server Version: 4.5.0-0.nightly-2020-08-01-204100 Kubernetes Version: v1.18.3+08c38ef OCS verison: cluster version NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.0-0.nightly-2020-08-01-204100 True False 139m Cluster version is 4.5.0-0.nightly-2020-08-01-204100 Rook version Ceph version OCP version: Client Version: 4.3.8 Server Version: 4.5.0-0.nightly-2020-08-31-101523 Kubernetes Version: v1.18.3+6c42de8 OCS verison: ocs-operator.v4.5.0-67.ci OpenShift Container Storage 4.5.0-67.ci Succeeded cluster version NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.0-0.nightly-2020-08-31-101523 True False 3h14m Cluster version is 4.5.0-0.nightly-2020-08-31-101523 Rook version rook: 4.5-46.d1b1f29c.release_4.5 go: go1.13.4 Ceph version ceph version 14.2.8-91.el8cp (75b4845da7d469665bd48d1a49badcc3677bf5cd) nautilus (stable) Created attachment 1713436 [details]
node replacement logs AWS UPI
|