Fedora Account System
Red Hat Associate
Red Hat Customer
Drive replacement instructions change for removing the symlink step Reported by: aclewett https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.4/html/managing_openshift_container_storage/replacing-storage-devices_rhocs#annotations:29a591f7-eb35-41c6-92ea-ff07de3ce2dc
In the drive replacement process the symlink must be removed before new PV is created using LSO with new by-id in the localvolume CR. This current documented process will fail if new drive has same device name as old drive (i.e. same slot).
Should also delete PV associated with PVC that was deleted in earlier step. This step, deleting PV, is much later in the process. If PV is deleted then the only thing to do after recreating new PV with new device (or same device reseated) is to look for a new available PV.
In documentation - https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.4/html/managing_openshift_container_storage/replacing-storage-devices_rhocs#replacing-operational-or-failed-storage-devices-on-vmware-and-bare-metal-infrastructures_rhocs - Step 7 and Step 8 need to be reversed when removing 'bad' drive from a hot pluggable slot and inserting new drive into same slot.
Content now in new KCS that needs to replace current content in section https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.4/html/managing_openshift_container_storage/replacing-storage-devices_rhocs#replacing-operational-or-failed-storage-devices-on-vmware-and-bare-metal-infrastructures_rhocs. KCS with new content - https://access.redhat.com/solutions/4982091 Markdown version - https://github.com/red-hat-storage/ocs-training/blob/master/ocp4ocs4/device-replacement.md
Tested on BM LSO environment Version: OCP: 4.5.0-0.nightly-2020-08-23-191713 OCS: ocs-operator.v4.5.0-54.ci Followed the document [1] for device replcament. Small observations: - In step5: Verify that the OSD is removed successfully by checking the ocs-osd-removal pod logs. The provided command didn't work for me and the below command worked: $ oc get pods -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage NAME READY STATUS RESTARTS AGE ocs-osd-removal-0-jdmx8 0/1 Completed 0 22s - And also in note same step: oc logs ${osd_id_to_remove} -n openshift-storage --tail=-1, didn't work Comparing both 4.5[1] & 4.4[2] documents, small observation: - In step 2, Scale down the OSD deployment for the OSD to be replaced - in 4.4 document we are missing explanation about osd_id_to_remove. - In step 4, Remove the old OSD from the cluster so that a new OSD can be added. - in 4.4 document we are missing, 4.i step, i.e Delete any old ocs-osd-removal jobs. - In step 5, Verify that the OSD is removed successfully by checking the ocs-osd-removal pod logs. - Both document has differnt way to verify. Is this expected behaviour? - In step 7, Replace the old device and use the new device to create a new OpenShift Container Platform PV. - We see difference in both document, in 4.4 we are removing and adding disk in the same step and next step we are removing symlink. From the comment13, afaiu we need to backport changes to 4.4 too. Am I missing anything here? [1]: https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.5/html-single/managing_Inopenshift_container_storage/index?lb_target=preview#replacing-a-local-storage-device_rhocs [2]: https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.4/html-single/managing_openshift_container_storage/index?lb_target=preview#replacing-operational-or-failed-storage-devices-on-vmware-and-bare-metal-infrastructures_rhocs Steps performed: http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/bz-1854614/commands_performed
One more observations, missed in previous comment. In 4.5 document, 7th step : Record the /dev/disk/by-id/{id} that is to be replaced using the device name, sdb, identified earlier. - here we are referring to nvme in examples but the earlier it was referring to as sdb.
Tested on VMWARE environment Version: OCP: 4.5.0-0.nightly-2020-08-23-191713 OCS: ocs-operator.v4.5.0-64.ci Followed the document [1] for device replacement and successfully able to replace a device. Thanks for all the changes. Looks good to me just a small change required on step 5. - step 5: Verify that the OSD is removed successfully by checking the status of the ocs-osd-removal pod. A status of Completed confirms that the OSD removal job succeeded. - in note: If ocs-osd-removal fails and the pod is not in the expected Completed state, check the pod logs for further debugging. For example: # oc logs ${osd_id_to_remove} -n openshift-storage --tail=-1 This command doesn't work for me. $ oc logs ${osd_id_to_remove} -n openshift-storage --tail=-1 Error from server (NotFound): pods "0" not found [1]: https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.5/html-single/managing_openshift_container_storage/index?lb_target=preview steps-performed: http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/bz-1854614/re-validation/steps_perfromed
Thank you Erin. Confirmed the content from both 4.5 and 4.4 docs changes are present, moving the bz to verified state. # oc logs -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage --tail=-1