Description of problem: [cephadm] 5.0 - OSD replacement feature - Preserved OSD ID's is not getting allocated for the newly added OSD device Version-Release number of selected component (if applicable): [root@ceph-adm7 ~]# sudo cephadm version Using recent ceph image registry.redhat.io/rhceph-alpha/rhceph-5-rhel8:latest ceph version 16.0.0-7953.el8cp (aac7c5c7d5f82d2973c366730f65255afd66e515) pacific (dev) How reproducible: Steps to Reproduce: 1. Install 5.0 cluster with dashboard enabled 2. Enter to cephadm shell 3. check ceph status and make sure all OSDs are up and IN state 4. Followed the below steps a)ceph osd tree --> check all OSds b) Ceph orch osd rm 4 --> remove 4 c) ceph osd tree --> 4 should been removed d) ceph orch device zap ceph-adm7 /dev/sdb --force --> clear the data e) ceph orch device ls ---> Device should be available to reuse for adding new disk f) Ceph orch osd rm 1 --replace --> preserving OSD ID 1 g) ceph orch daemon osd add ceph-adm7:/dev/sdb/ --adding new disk to get the preserved OSD ID h)ceph osd tree --> observe the behaviour Actual results: Preserve ID 1 is not been allocated instead the removed OSD ID 4 is seen in ceph OSD tree Expected results: OSD ID 1 should be added when we add newly disk Additional info: 10.74.253.36 root/redhat output: ********************************************************************************************** ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.34175 root default -7 0.07808 host ceph-adm7 4 hdd 0.02930 osd.4 up 1.00000 1.00000 12 hdd 0.02930 osd.12 up 1.00000 1.00000 13 hdd 0.01949 osd.13 up 1.00000 1.00000 -3 0.14648 host ceph-adm8 1 hdd 0.05859 osd.1 up 1.00000 1.00000 2 hdd 0.02930 osd.2 up 1.00000 1.00000 6 hdd 0.02930 osd.6 up 1.00000 1.00000 8 hdd 0.02930 osd.8 up 1.00000 1.00000 -5 0.11719 host ceph-adm9 3 hdd 0.02930 osd.3 destroyed 0 1.00000 5 hdd 0.02930 osd.5 destroyed 0 1.00000 7 hdd 0.02930 osd.7 up 1.00000 1.00000 9 hdd 0.02930 osd.9 destroyed 0 1.00000 0 0 osd.0 down 0 1.00000 [ceph: root@ceph-adm7 /]# ceph device ls DEVICE HOST:DEV DAEMONS LIFE EXPECTANCY QEMU_QEMU_HARDDISK_073ed4af-4752-4956-9e09-6504da882a79 ceph-adm9:sdf osd.0 QEMU_QEMU_HARDDISK_3bc5e11c-28f2-419e-b076-8b6032e49de5 ceph-adm8:sdc osd.2 QEMU_QEMU_HARDDISK_46e30862-254f-4327-bb00-99ea29f8e237 ceph-adm8:sdf osd.1 QEMU_QEMU_HARDDISK_572813c0-bce4-46f0-a388-bb9ba92a4c9c ceph-adm8:sde osd.8 QEMU_QEMU_HARDDISK_5df4866b-18c5-4de5-8ce1-f44084b67e74 ceph-adm7:sdf mon.ceph-adm7 QEMU_QEMU_HARDDISK_5eed3652-9334-408b-b0e7-3a6d125a7acc ceph-adm7:sdb osd.4 QEMU_QEMU_HARDDISK_6a660612-aa36-4e56-a80f-01839475e55d ceph-adm7:sde osd.13 QEMU_QEMU_HARDDISK_7c92121d-7ee6-4545-9820-14449e78892c ceph-adm9:sdb osd.3 QEMU_QEMU_HARDDISK_7e0b094b-662c-4320-82af-353c993e46bb ceph-adm9:sda mon.ceph-adm9 QEMU_QEMU_HARDDISK_bb888a81-55a6-4418-a9e5-c79043d1bbf7 ceph-adm7:sdd osd.12 QEMU_QEMU_HARDDISK_d81b73c5-ab55-4a41-9f77-81533496ac16 ceph-adm9:sdd osd.7 QEMU_QEMU_HARDDISK_ee309705-a09e-4e31-83e7-3b380398f255 ceph-adm8:sda mon.ceph-adm8 QEMU_QEMU_HARDDISK_f65c4443-18fb-4d02-917d-6a6761541dab ceph-adm8:sdd osd.6 QEMU_QEMU_HARDDISK_ff170b5d-c13f-4514-9685-532e3b3c798e ceph-adm9:sdc osd.5 QEMU_QEMU_HARDDISK_ff2f239d-9870-4ee9-b7a1-20d01ad318cc ceph-adm9:sde osd.9 [ceph: root@ceph-adm7 /]# ceph orch osd rm 4 Scheduled OSD(s) for removal [ceph: root@ceph-adm7 /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.31245 root default -7 0.04878 host ceph-adm7 12 hdd 0.02930 osd.12 up 1.00000 1.00000 13 hdd 0.01949 osd.13 up 1.00000 1.00000 -3 0.14648 host ceph-adm8 1 hdd 0.05859 osd.1 up 1.00000 1.00000 2 hdd 0.02930 osd.2 up 1.00000 1.00000 6 hdd 0.02930 osd.6 up 1.00000 1.00000 8 hdd 0.02930 osd.8 up 1.00000 1.00000 -5 0.11719 host ceph-adm9 3 hdd 0.02930 osd.3 destroyed 0 1.00000 5 hdd 0.02930 osd.5 destroyed 0 1.00000 7 hdd 0.02930 osd.7 up 0 1.00000 9 hdd 0.02930 osd.9 destroyed 0 1.00000 0 0 osd.0 down 0 1.00000 [ceph: root@ceph-adm7 /]# [ceph: root@ceph-adm7 /]# ceph orch device zap ceph-adm7 /dev/sdb --force /bin/podman:stderr WARNING: The same type, major and minor should not be used for multiple devices. /bin/podman:stderr --> Zapping: /dev/sdb /bin/podman:stderr --> Zapping lvm member /dev/sdb. lv_path is /dev/ceph-7b04f4e5-ea07-442b-93f2-6f6d96276108/osd-block-b11b448e-60cd-4bde-b6ea-ac76a5284b9b /bin/podman:stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-7b04f4e5-ea07-442b-93f2-6f6d96276108/osd-block-b11b448e-60cd-4bde-b6ea-ac76a5284b9b bs=1M count=10 conv=fsync /bin/podman:stderr stderr: 10+0 records in /bin/podman:stderr 10+0 records out /bin/podman:stderr 10485760 bytes (10 MB, 10 MiB) copied, 0.0383926 s, 273 MB/s /bin/podman:stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-7b04f4e5-ea07-442b-93f2-6f6d96276108 /bin/podman:stderr Running command: /usr/sbin/vgremove -v -f ceph-7b04f4e5-ea07-442b-93f2-6f6d96276108 /bin/podman:stderr stderr: Removing ceph--7b04f4e5--ea07--442b--93f2--6f6d96276108-osd--block--b11b448e--60cd--4bde--b6ea--ac76a5284b9b (253:6) /bin/podman:stderr stderr: Archiving volume group "ceph-7b04f4e5-ea07-442b-93f2-6f6d96276108" metadata (seqno 5). /bin/podman:stderr stderr: Releasing logical volume "osd-block-b11b448e-60cd-4bde-b6ea-ac76a5284b9b" /bin/podman:stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-7b04f4e5-ea07-442b-93f2-6f6d96276108" (seqno 6). /bin/podman:stderr stdout: Logical volume "osd-block-b11b448e-60cd-4bde-b6ea-ac76a5284b9b" successfully removed /bin/podman:stderr stderr: Removing physical volume "/dev/sdb" from volume group "ceph-7b04f4e5-ea07-442b-93f2-6f6d96276108" /bin/podman:stderr stdout: Volume group "ceph-7b04f4e5-ea07-442b-93f2-6f6d96276108" successfully removed /bin/podman:stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M count=10 conv=fsync /bin/podman:stderr stderr: 10+0 records in /bin/podman:stderr 10+0 records out /bin/podman:stderr 10485760 bytes (10 MB, 10 MiB) copied, 0.0234263 s, 448 MB/s /bin/podman:stderr --> Zapping successful for: <Raw Device: /dev/sdb> [ceph: root@ceph-adm7 /]# ceph device ls DEVICE HOST:DEV DAEMONS LIFE EXPECTANCY QEMU_QEMU_HARDDISK_073ed4af-4752-4956-9e09-6504da882a79 ceph-adm9:sdf osd.0 QEMU_QEMU_HARDDISK_3bc5e11c-28f2-419e-b076-8b6032e49de5 ceph-adm8:sdc osd.2 QEMU_QEMU_HARDDISK_46e30862-254f-4327-bb00-99ea29f8e237 ceph-adm8:sdf osd.1 QEMU_QEMU_HARDDISK_572813c0-bce4-46f0-a388-bb9ba92a4c9c ceph-adm8:sde osd.8 QEMU_QEMU_HARDDISK_5df4866b-18c5-4de5-8ce1-f44084b67e74 ceph-adm7:sdf mon.ceph-adm7 QEMU_QEMU_HARDDISK_6a660612-aa36-4e56-a80f-01839475e55d ceph-adm7:sde osd.13 QEMU_QEMU_HARDDISK_7c92121d-7ee6-4545-9820-14449e78892c ceph-adm9:sdb osd.3 QEMU_QEMU_HARDDISK_7e0b094b-662c-4320-82af-353c993e46bb ceph-adm9:sda mon.ceph-adm9 QEMU_QEMU_HARDDISK_bb888a81-55a6-4418-a9e5-c79043d1bbf7 ceph-adm7:sdd osd.12 QEMU_QEMU_HARDDISK_d81b73c5-ab55-4a41-9f77-81533496ac16 ceph-adm9:sdd osd.7 QEMU_QEMU_HARDDISK_ee309705-a09e-4e31-83e7-3b380398f255 ceph-adm8:sda mon.ceph-adm8 QEMU_QEMU_HARDDISK_f65c4443-18fb-4d02-917d-6a6761541dab ceph-adm8:sdd osd.6 QEMU_QEMU_HARDDISK_ff170b5d-c13f-4514-9685-532e3b3c798e ceph-adm9:sdc osd.5 QEMU_QEMU_HARDDISK_ff2f239d-9870-4ee9-b7a1-20d01ad318cc ceph-adm9:sde osd.9 [ceph: root@ceph-adm7 /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.31245 root default -7 0.04878 host ceph-adm7 12 hdd 0.02930 osd.12 up 1.00000 1.00000 13 hdd 0.01949 osd.13 up 1.00000 1.00000 -3 0.14648 host ceph-adm8 1 hdd 0.05859 osd.1 up 1.00000 1.00000 2 hdd 0.02930 osd.2 up 1.00000 1.00000 6 hdd 0.02930 osd.6 up 1.00000 1.00000 8 hdd 0.02930 osd.8 up 1.00000 1.00000 -5 0.11719 host ceph-adm9 3 hdd 0.02930 osd.3 destroyed 0 1.00000 5 hdd 0.02930 osd.5 destroyed 0 1.00000 7 hdd 0.02930 osd.7 up 0 1.00000 9 hdd 0.02930 osd.9 destroyed 0 1.00000 0 0 osd.0 down 0 1.00000 [ceph: root@ceph-adm7 /]# ceph orch osd rm 1 --replace Scheduled OSD(s) for removal [ceph: root@ceph-adm7 /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.31245 root default -7 0.04878 host ceph-adm7 12 hdd 0.02930 osd.12 up 1.00000 1.00000 13 hdd 0.01949 osd.13 up 1.00000 1.00000 -3 0.14648 host ceph-adm8 1 hdd 0.05859 osd.1 destroyed 1.00000 1.00000 2 hdd 0.02930 osd.2 up 1.00000 1.00000 6 hdd 0.02930 osd.6 up 1.00000 1.00000 8 hdd 0.02930 osd.8 up 1.00000 1.00000 -5 0.11719 host ceph-adm9 3 hdd 0.02930 osd.3 destroyed 0 1.00000 5 hdd 0.02930 osd.5 destroyed 0 1.00000 7 hdd 0.02930 osd.7 up 0 1.00000 9 hdd 0.02930 osd.9 destroyed 0 1.00000 0 0 osd.0 down 0 1.00000 [ceph: root@ceph-adm7 /]# [ceph: root@ceph-adm7 /]# #ceph orch device zap ceph-adm7 /dev/sdb --force [ceph: root@ceph-adm7 /]# ceph orch daemon add osd ceph-adm7:/dev/sdb Created osd(s) 4 on host 'ceph-adm7' [ceph: root@ceph-adm7 /]# [ceph: root@ceph-adm7 /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.34175 root default -7 0.07808 host ceph-adm7 4 hdd 0.02930 osd.4 up 1.00000 1.00000 12 hdd 0.02930 osd.12 up 1.00000 1.00000 13 hdd 0.01949 osd.13 up 1.00000 1.00000 -3 0.14648 host ceph-adm8 1 hdd 0.05859 osd.1 destroyed 1.00000 1.00000 2 hdd 0.02930 osd.2 up 1.00000 1.00000 6 hdd 0.02930 osd.6 up 1.00000 1.00000 8 hdd 0.02930 osd.8 up 1.00000 1.00000 -5 0.11719 host ceph-adm9 3 hdd 0.02930 osd.3 destroyed 0 1.00000 5 hdd 0.02930 osd.5 destroyed 0 1.00000 7 hdd 0.02930 osd.7 up 0 1.00000 9 hdd 0.02930 osd.9 destroyed 0 1.00000 0 0 osd.0 down 0 1.00000 [ceph: root@ceph-adm7 /]# [ceph: root@ceph-adm7 /]# ceph orch ls NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME IMAGE ID alertmanager 1/1 54s ago 2w count:1 registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.5 b7bae610cd46 crash 3/3 4m ago 2w * registry.redhat.io/rhceph-alpha/rhceph-5-rhel8:latest c88a5d60f510 grafana 1/1 54s ago 2w count:1 registry.redhat.io/rhceph-alpha/rhceph-5-dashboard-rhel8:latest bd3d7748747b mgr 3/2 4m ago 8d <unmanaged> registry.redhat.io/rhceph-alpha/rhceph-5-rhel8:latest c88a5d60f510 mon 3/3 4m ago 2w ceph-adm7;ceph-adm8;ceph-adm9 registry.redhat.io/rhceph-alpha/rhceph-5-rhel8:latest c88a5d60f510 node-exporter 1/3 4m ago 2w * registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5 mix osd.None 2/0 2m ago - <unmanaged> registry.redhat.io/rhceph-alpha/rhceph-5-rhel8:latest c88a5d60f510 osd.all-available-devices 6/6 4m ago 5h <unmanaged> registry.redhat.io/rhceph-alpha/rhceph-5-rhel8:latest c88a5d60f510 prometheus 1/1 54s ago 2w count:1 registry.redhat.io/openshift4/ose-prometheus:v4.6 bebb0ddef7f0 [ceph: root@ceph-adm7 /]#
@Preethi: The reuse of the osd id <x> only takes place in the same host where the osd x was deleted. ( when you show me the problem i did not realize that we were trying to reuse the id in a different host) The use case for this feature is to make more easy and avoid data movement between cluster hosts when we are trying to replace a damaged PHYSICAL storage device. The new OSD will be created always in the same host (because we are replacing a device), but using a new storage device with the "preserved" osd id. the sequence of operations needed to reproduce the use case will be: Example 1. Start point: ================ - A node with 3 devices used by 3 osds. - No osd managed services that could affect free devices in the node ( to avoid getting this devices used automatically as soon as the devices are available ) [ceph: root@cephLab2-node-00 /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -7 0.14639 host cephLab2-node-02 2 hdd 0.04880 osd.2 up 1.00000 1.00000 <--- /dev/vdb 5 hdd 0.04880 osd.5 up 1.00000 1.00000 <--- /dev/vdc 7 hdd 0.04880 osd.7 up 1.00000 1.00000 <--- /dev/vdd [ceph: root@cephLab2-node-00 /]# ceph orch ls osd.None 2/0 7m ago - <unmanaged> .... <--- UNMANAGED osd.all-available-devices 6/6 7m ago 99m <unmanaged> .... <--- UNMANAGED 2. Prepare a free device to be used with a new osd ( if you have "available devices" in the host , you can go over this step) ============================================================================================================================= [ceph: root@cephLab2-node-00 /]# ceph orch osd rm 2 Scheduled OSD(s) for removal [ceph: root@cephLab2-node-00 /]# ceph orch osd rm status No OSD remove/replace operations reported [ceph: root@cephLab2-node-00 /]# ceph orch device zap cephLab2-node-02 /dev/vdb --force ... /bin/podman: --> Zapping successful for: <Raw Device: /dev/vdb> [ceph: root@cephLab2-node-00 /]# ceph orch device ls Hostname Path Type Serial Size Health Ident Fault Available ... cephLab2-node-02 /dev/vdb hdd 53.6G Unknown N/A N/A Yes <---- we have the available device for new osds ... [ceph: root@cephLab2-node-00 /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF ... -7 0.09760 host cephLab2-node-02 <---- OSD 2 deleted 5 hdd 0.04880 osd.5 up 1.00000 1.00000 7 hdd 0.04880 osd.7 up 1.00000 1.00000 3. Remove one OSD preserving the osd id ======================================== Lets suppose that the disk backing up osd.7 has been attacked by the silicon moth, and does not work any more. You can REPLACE osd.7 using your brand and new available "/dev/vdb" device: [ceph: root@cephLab2-node-00 /]# ceph orch osd rm 7 --replace Scheduled OSD(s) for removal [ceph: root@cephLab2-node-00 /]# ceph orch osd rm status No OSD remove/replace operations reported NOTE: I do not zap the disk... not needed, because the silicon moth have caused a complete destruction and this device is not going to be available never more. [ceph: root@cephLab2-node-00 /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF ... -7 0.09760 host cephLab2-node-02 5 hdd 0.04880 osd.5 up 1.00000 1.00000 7 hdd 0.04880 osd.7 destroyed 0 1.00000 <----- This means that osd id 7 is preserved to be reused with the first osd creation that will happen in this host. [ceph: root@cephLab2-node-00 /]# ceph orch device ls Hostname Path Type Serial Size Health Ident Fault Available ... cephLab2-node-02 /dev/vdb hdd 53.6G Unknown N/A N/A Yes <--- our new brand device cephLab2-node-02 /dev/vdc hdd 53.6G Unknown N/A N/A No cephLab2-node-02 /dev/vdd hdd 53.6G Unknown N/A N/A No <----attacked by silicon moth.. not usable 4. Create a new osd using the preserved id =========================================== We will use the available device "cephLab2-node-02 /dev/vdb" to create a new osd that will become osd.7 ( the preserved osd id ) [ceph: root@cephLab2-node-00 /]# ceph orch daemon add osd cephLab2-node-02:/dev/vdb Created osd(s) 7 on host 'cephLab2-node-02' <---- it has used the osd id 7!! Viva! [ceph: root@cephLab2-node-00 /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF ... -7 0.09760 host cephLab2-node-02 5 hdd 0.04880 osd.5 up 1.00000 1.00000 7 hdd 0.04880 osd.7 up 1.00000 1.00000 [ceph: root@cephLab2-node-00 /]# ceph orch device ls Hostname Path Type Serial Size Health Ident Fault Available ... cephLab2-node-02 /dev/vdb hdd 53.6G Unknown N/A N/A No cephLab2-node-02 /dev/vdc hdd 53.6G Unknown N/A N/A No cephLab2-node-02 /dev/vdd hdd 53.6G Unknown N/A N/A No <---- destroyed by silicon moth..physical device 5. Replace the damaged physical device ======================================= If you have money and time, now you can put the host in maintenance mode and replace the damaged device. Once the device will be replaced, a new available device will be ready for new osds.
Hi Juan, Thanks for the detailed steps and explanation. I do not see any issue now. Preserve OSD IDs are getting allocated to the newly added device. Below output for reference. [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch ls NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME IMAGE ID alertmanager 2/2 6m ago 17h ceph-sunil1adm-1614692246522-node2-mon-mds-node-exporter-alertm;ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.5 3ea01d72d22c crash 0/8 - - <unmanaged> <unknown> <unknown> grafana 1/1 5m ago 17h ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor registry.redhat.io/rhceph-alpha/rhceph-5-dashboard-rhel8:latest bd3d7748747b iscsi.iscsi 1/1 7m ago 17h ceph-sunil1adm-1614692246522-node8-client-nfs-node-exporter-isc registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:687d060d91102d9317fbd8ec0305a24f53bdfb86d7ca3aaacc664955da01f03f 700feae6f592 mds.cephfs 2/2 7m ago 17h ceph-sunil1adm-1614692246522-node2-mon-mds-node-exporter-alertm;ceph-sunil1adm-1614692246522-node8-client-nfs-node-exporter-isc;count:2 registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:687d060d91102d9317fbd8ec0305a24f53bdfb86d7ca3aaacc664955da01f03f 700feae6f592 mgr 1/1 5m ago 18h ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-5.0-rhel-8-containers-candidate-21981-20210302003306 700feae6f592 mon 3/3 7m ago 16h label:mon mix 700feae6f592 node-exporter 8/8 8m ago 17h * registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5 a6af8f87dd4a osd.all-available-devices 12/12 8m ago 4s <unmanaged> registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:687d060d91102d9317fbd8ec0305a24f53bdfb86d7ca3aaacc664955da01f03f 700feae6f592 prometheus 1/1 5m ago 17h ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor registry.redhat.io/openshift4/ose-prometheus:v4.6 6050e785b668 [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.17505 root default -7 0.05835 host ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash 2 hdd 0.01459 osd.2 up 1.00000 1.00000 5 hdd 0.01459 osd.5 up 1.00000 1.00000 8 hdd 0.01459 osd.8 up 1.00000 1.00000 11 hdd 0.01459 osd.11 up 1.00000 1.00000 -3 0.05835 host ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash 0 hdd 0.01459 osd.0 up 1.00000 1.00000 3 hdd 0.01459 osd.3 up 1.00000 1.00000 6 hdd 0.01459 osd.6 up 1.00000 1.00000 9 hdd 0.01459 osd.9 up 1.00000 1.00000 -5 0.05835 host ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash 1 hdd 0.01459 osd.1 up 1.00000 1.00000 4 hdd 0.01459 osd.4 up 1.00000 1.00000 7 hdd 0.01459 osd.7 up 1.00000 1.00000 10 hdd 0.01459 osd.10 up 1.00000 1.00000 [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch osd rm 2^C [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph device ls DEVICE HOST:DEV DAEMONS LIFE EXPECTANCY 06de0da0-7fe2-4892-8 ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash:vde osd.9 3560e149-2138-414a-b ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash:vdd osd.8 4e463e20-2240-4364-9 ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash:vdd osd.7 54f8bc06-a6d5-4979-b ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash:vdc osd.4 61bbf7df-e0f4-4dd9-8 ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash:vdb osd.1 6841af0b-a115-4360-b ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash:vde osd.10 7f12e9dc-bb16-48db-9 ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash:vdd osd.6 804acaa8-a5ff-4af2-8 ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash:vdc osd.5 842037ad-6533-4978-9 ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash:vde osd.11 cc17dfbf-b177-46db-9 ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash:vdc osd.3 e1753d7b-cc8b-408a-9 ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash:vdb osd.2 e4f0f5b9-888e-40ba-b ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash:vdb osd.0 [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch device ls Hostname Path Type Serial Size Health Ident Fault Available ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash /dev/vdb hdd e1753d7b-cc8b-408a-9 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash /dev/vdc hdd 804acaa8-a5ff-4af2-8 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash /dev/vdd hdd 3560e149-2138-414a-b 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash /dev/vde hdd 842037ad-6533-4978-9 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash /dev/vdb hdd e4f0f5b9-888e-40ba-b 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash /dev/vdc hdd cc17dfbf-b177-46db-9 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash /dev/vdd hdd 7f12e9dc-bb16-48db-9 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash /dev/vde hdd 06de0da0-7fe2-4892-8 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash /dev/vdb hdd 61bbf7df-e0f4-4dd9-8 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash /dev/vdc hdd 54f8bc06-a6d5-4979-b 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash /dev/vdd hdd 4e463e20-2240-4364-9 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash /dev/vde hdd 6841af0b-a115-4360-b 16.1G Unknown N/A N/A No [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch osd rm 2 Scheduled OSD(s) for removal [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.16046 root default -7 0.04376 host ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash 5 hdd 0.01459 osd.5 up 1.00000 1.00000 8 hdd 0.01459 osd.8 up 1.00000 1.00000 11 hdd 0.01459 osd.11 up 1.00000 1.00000 -3 0.05835 host ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash 0 hdd 0.01459 osd.0 up 1.00000 1.00000 3 hdd 0.01459 osd.3 up 1.00000 1.00000 6 hdd 0.01459 osd.6 up 1.00000 1.00000 9 hdd 0.01459 osd.9 up 1.00000 1.00000 -5 0.05835 host ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash 1 hdd 0.01459 osd.1 up 1.00000 1.00000 4 hdd 0.01459 osd.4 up 1.00000 1.00000 7 hdd 0.01459 osd.7 up 1.00000 1.00000 10 hdd 0.01459 osd.10 up 1.00000 1.00000 [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph osd ^C [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch osd rm status No OSD remove/replace operations reported [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch device zap ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash /dev/vdb --force /bin/podman: WARNING: The same type, major and minor should not be used for multiple devices. /bin/podman: WARNING: The same type, major and minor should not be used for multiple devices. /bin/podman: WARNING: The same type, major and minor should not be used for multiple devices. /bin/podman: WARNING: The same type, major and minor should not be used for multiple devices. /bin/podman: --> Zapping: /dev/vdb /bin/podman: --> Zapping lvm member /dev/vdb. lv_path is /dev/ceph-94a5d8e2-7bbd-4a87-b6da-38fc7290a12c/osd-block-054454e1-e7d0-4832-8347-50914f2b4b02 /bin/podman: Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-94a5d8e2-7bbd-4a87-b6da-38fc7290a12c/osd-block-054454e1-e7d0-4832-8347-50914f2b4b02 bs=1M count=10 conv=fsync /bin/podman: stderr: 10+0 records in /bin/podman: 10+0 records out /bin/podman: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0600459 s, 175 MB/s /bin/podman: --> Only 1 LV left in VG, will proceed to destroy volume group ceph-94a5d8e2-7bbd-4a87-b6da-38fc7290a12c /bin/podman: Running command: /usr/sbin/vgremove -v -f ceph-94a5d8e2-7bbd-4a87-b6da-38fc7290a12c /bin/podman: stderr: Removing ceph--94a5d8e2--7bbd--4a87--b6da--38fc7290a12c-osd--block--054454e1--e7d0--4832--8347--50914f2b4b02 (253:0) /bin/podman: stderr: Archiving volume group "ceph-94a5d8e2-7bbd-4a87-b6da-38fc7290a12c" metadata (seqno 5). /bin/podman: Releasing logical volume "osd-block-054454e1-e7d0-4832-8347-50914f2b4b02" /bin/podman: stderr: Creating volume group backup "/etc/lvm/backup/ceph-94a5d8e2-7bbd-4a87-b6da-38fc7290a12c" (seqno 6). /bin/podman: stdout: Logical volume "osd-block-054454e1-e7d0-4832-8347-50914f2b4b02" successfully removed /bin/podman: stderr: Removing physical volume "/dev/vdb" from volume group "ceph-94a5d8e2-7bbd-4a87-b6da-38fc7290a12c" /bin/podman: stdout: Volume group "ceph-94a5d8e2-7bbd-4a87-b6da-38fc7290a12c" successfully removed /bin/podman: Running command: /usr/bin/dd if=/dev/zero of=/dev/vdb bs=1M count=10 conv=fsync /bin/podman: --> Zapping successful for: <Raw Device: /dev/vdb> [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch device ls Hostname Path Type Serial Size Health Ident Fault Available ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash /dev/vdb hdd e1753d7b-cc8b-408a-9 16.1G Unknown N/A N/A Yes ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash /dev/vdc hdd 804acaa8-a5ff-4af2-8 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash /dev/vdd hdd 3560e149-2138-414a-b 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash /dev/vde hdd 842037ad-6533-4978-9 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash /dev/vdb hdd e4f0f5b9-888e-40ba-b 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash /dev/vdc hdd cc17dfbf-b177-46db-9 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash /dev/vdd hdd 7f12e9dc-bb16-48db-9 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash /dev/vde hdd 06de0da0-7fe2-4892-8 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash /dev/vdb hdd 61bbf7df-e0f4-4dd9-8 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash /dev/vdc hdd 54f8bc06-a6d5-4979-b 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash /dev/vdd hdd 4e463e20-2240-4364-9 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash /dev/vde hdd 6841af0b-a115-4360-b 16.1G Unknown N/A N/A No [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.16046 root default -7 0.04376 host ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash 5 hdd 0.01459 osd.5 up 1.00000 1.00000 8 hdd 0.01459 osd.8 up 1.00000 1.00000 11 hdd 0.01459 osd.11 up 1.00000 1.00000 -3 0.05835 host ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash 0 hdd 0.01459 osd.0 up 1.00000 1.00000 3 hdd 0.01459 osd.3 up 1.00000 1.00000 6 hdd 0.01459 osd.6 up 1.00000 1.00000 9 hdd 0.01459 osd.9 up 1.00000 1.00000 -5 0.05835 host ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash 1 hdd 0.01459 osd.1 up 1.00000 1.00000 4 hdd 0.01459 osd.4 up 1.00000 1.00000 7 hdd 0.01459 osd.7 up 1.00000 1.00000 10 hdd 0.01459 osd.10 up 1.00000 1.00000 [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch osd rm 5 --replace Scheduled OSD(s) for removal [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch osd rm status No OSD remove/replace operations reported [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.16046 root default -7 0.04376 host ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash 5 hdd 0.01459 osd.5 destroyed 0 1.00000 8 hdd 0.01459 osd.8 up 1.00000 1.00000 11 hdd 0.01459 osd.11 up 1.00000 1.00000 -3 0.05835 host ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash 0 hdd 0.01459 osd.0 up 1.00000 1.00000 3 hdd 0.01459 osd.3 up 1.00000 1.00000 6 hdd 0.01459 osd.6 up 1.00000 1.00000 9 hdd 0.01459 osd.9 up 1.00000 1.00000 -5 0.05835 host ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash 1 hdd 0.01459 osd.1 up 1.00000 1.00000 4 hdd 0.01459 osd.4 up 1.00000 1.00000 7 hdd 0.01459 osd.7 up 1.00000 1.00000 10 hdd 0.01459 osd.10 up 1.00000 1.00000 [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch daemon add osd ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash:/dev/vdb Created osd(s) 5 on host 'ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash' [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.16046 root default -7 0.04376 host ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash 5 hdd 0.01459 osd.5 up 1.00000 1.00000 8 hdd 0.01459 osd.8 up 1.00000 1.00000 11 hdd 0.01459 osd.11 up 1.00000 1.00000 -3 0.05835 host ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash 0 hdd 0.01459 osd.0 up 1.00000 1.00000 3 hdd 0.01459 osd.3 up 1.00000 1.00000 6 hdd 0.01459 osd.6 up 1.00000 1.00000 9 hdd 0.01459 osd.9 up 1.00000 1.00000 -5 0.05835 host ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash 1 hdd 0.01459 osd.1 up 1.00000 1.00000 4 hdd 0.01459 osd.4 up 1.00000 1.00000 7 hdd 0.01459 osd.7 up 1.00000 1.00000 10 hdd 0.01459 osd.10 up 1.00000 1.00000 [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]#
Hi Juan, Thanks for the detailed steps and explanation. I do not see any issue now. Preserve OSD IDs are getting allocated to the newly added device. Below output for reference. [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch ls NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME IMAGE ID alertmanager 2/2 6m ago 17h ceph-sunil1adm-1614692246522-node2-mon-mds-node-exporter-alertm;ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.5 3ea01d72d22c crash 0/8 - - <unmanaged> <unknown> <unknown> grafana 1/1 5m ago 17h ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor registry.redhat.io/rhceph-alpha/rhceph-5-dashboard-rhel8:latest bd3d7748747b iscsi.iscsi 1/1 7m ago 17h ceph-sunil1adm-1614692246522-node8-client-nfs-node-exporter-isc registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:687d060d91102d9317fbd8ec0305a24f53bdfb86d7ca3aaacc664955da01f03f 700feae6f592 mds.cephfs 2/2 7m ago 17h ceph-sunil1adm-1614692246522-node2-mon-mds-node-exporter-alertm;ceph-sunil1adm-1614692246522-node8-client-nfs-node-exporter-isc;count:2 registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:687d060d91102d9317fbd8ec0305a24f53bdfb86d7ca3aaacc664955da01f03f 700feae6f592 mgr 1/1 5m ago 18h ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-5.0-rhel-8-containers-candidate-21981-20210302003306 700feae6f592 mon 3/3 7m ago 16h label:mon mix 700feae6f592 node-exporter 8/8 8m ago 17h * registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5 a6af8f87dd4a osd.all-available-devices 12/12 8m ago 4s <unmanaged> registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:687d060d91102d9317fbd8ec0305a24f53bdfb86d7ca3aaacc664955da01f03f 700feae6f592 prometheus 1/1 5m ago 17h ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor registry.redhat.io/openshift4/ose-prometheus:v4.6 6050e785b668 [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.17505 root default -7 0.05835 host ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash 2 hdd 0.01459 osd.2 up 1.00000 1.00000 5 hdd 0.01459 osd.5 up 1.00000 1.00000 8 hdd 0.01459 osd.8 up 1.00000 1.00000 11 hdd 0.01459 osd.11 up 1.00000 1.00000 -3 0.05835 host ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash 0 hdd 0.01459 osd.0 up 1.00000 1.00000 3 hdd 0.01459 osd.3 up 1.00000 1.00000 6 hdd 0.01459 osd.6 up 1.00000 1.00000 9 hdd 0.01459 osd.9 up 1.00000 1.00000 -5 0.05835 host ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash 1 hdd 0.01459 osd.1 up 1.00000 1.00000 4 hdd 0.01459 osd.4 up 1.00000 1.00000 7 hdd 0.01459 osd.7 up 1.00000 1.00000 10 hdd 0.01459 osd.10 up 1.00000 1.00000 [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch osd rm 2^C [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph device ls DEVICE HOST:DEV DAEMONS LIFE EXPECTANCY 06de0da0-7fe2-4892-8 ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash:vde osd.9 3560e149-2138-414a-b ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash:vdd osd.8 4e463e20-2240-4364-9 ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash:vdd osd.7 54f8bc06-a6d5-4979-b ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash:vdc osd.4 61bbf7df-e0f4-4dd9-8 ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash:vdb osd.1 6841af0b-a115-4360-b ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash:vde osd.10 7f12e9dc-bb16-48db-9 ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash:vdd osd.6 804acaa8-a5ff-4af2-8 ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash:vdc osd.5 842037ad-6533-4978-9 ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash:vde osd.11 cc17dfbf-b177-46db-9 ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash:vdc osd.3 e1753d7b-cc8b-408a-9 ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash:vdb osd.2 e4f0f5b9-888e-40ba-b ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash:vdb osd.0 [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch device ls Hostname Path Type Serial Size Health Ident Fault Available ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash /dev/vdb hdd e1753d7b-cc8b-408a-9 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash /dev/vdc hdd 804acaa8-a5ff-4af2-8 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash /dev/vdd hdd 3560e149-2138-414a-b 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash /dev/vde hdd 842037ad-6533-4978-9 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash /dev/vdb hdd e4f0f5b9-888e-40ba-b 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash /dev/vdc hdd cc17dfbf-b177-46db-9 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash /dev/vdd hdd 7f12e9dc-bb16-48db-9 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash /dev/vde hdd 06de0da0-7fe2-4892-8 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash /dev/vdb hdd 61bbf7df-e0f4-4dd9-8 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash /dev/vdc hdd 54f8bc06-a6d5-4979-b 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash /dev/vdd hdd 4e463e20-2240-4364-9 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash /dev/vde hdd 6841af0b-a115-4360-b 16.1G Unknown N/A N/A No [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch osd rm 2 Scheduled OSD(s) for removal [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.16046 root default -7 0.04376 host ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash 5 hdd 0.01459 osd.5 up 1.00000 1.00000 8 hdd 0.01459 osd.8 up 1.00000 1.00000 11 hdd 0.01459 osd.11 up 1.00000 1.00000 -3 0.05835 host ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash 0 hdd 0.01459 osd.0 up 1.00000 1.00000 3 hdd 0.01459 osd.3 up 1.00000 1.00000 6 hdd 0.01459 osd.6 up 1.00000 1.00000 9 hdd 0.01459 osd.9 up 1.00000 1.00000 -5 0.05835 host ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash 1 hdd 0.01459 osd.1 up 1.00000 1.00000 4 hdd 0.01459 osd.4 up 1.00000 1.00000 7 hdd 0.01459 osd.7 up 1.00000 1.00000 10 hdd 0.01459 osd.10 up 1.00000 1.00000 [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph osd ^C [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch osd rm status No OSD remove/replace operations reported [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch device zap ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash /dev/vdb --force /bin/podman: WARNING: The same type, major and minor should not be used for multiple devices. /bin/podman: WARNING: The same type, major and minor should not be used for multiple devices. /bin/podman: WARNING: The same type, major and minor should not be used for multiple devices. /bin/podman: WARNING: The same type, major and minor should not be used for multiple devices. /bin/podman: --> Zapping: /dev/vdb /bin/podman: --> Zapping lvm member /dev/vdb. lv_path is /dev/ceph-94a5d8e2-7bbd-4a87-b6da-38fc7290a12c/osd-block-054454e1-e7d0-4832-8347-50914f2b4b02 /bin/podman: Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-94a5d8e2-7bbd-4a87-b6da-38fc7290a12c/osd-block-054454e1-e7d0-4832-8347-50914f2b4b02 bs=1M count=10 conv=fsync /bin/podman: stderr: 10+0 records in /bin/podman: 10+0 records out /bin/podman: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0600459 s, 175 MB/s /bin/podman: --> Only 1 LV left in VG, will proceed to destroy volume group ceph-94a5d8e2-7bbd-4a87-b6da-38fc7290a12c /bin/podman: Running command: /usr/sbin/vgremove -v -f ceph-94a5d8e2-7bbd-4a87-b6da-38fc7290a12c /bin/podman: stderr: Removing ceph--94a5d8e2--7bbd--4a87--b6da--38fc7290a12c-osd--block--054454e1--e7d0--4832--8347--50914f2b4b02 (253:0) /bin/podman: stderr: Archiving volume group "ceph-94a5d8e2-7bbd-4a87-b6da-38fc7290a12c" metadata (seqno 5). /bin/podman: Releasing logical volume "osd-block-054454e1-e7d0-4832-8347-50914f2b4b02" /bin/podman: stderr: Creating volume group backup "/etc/lvm/backup/ceph-94a5d8e2-7bbd-4a87-b6da-38fc7290a12c" (seqno 6). /bin/podman: stdout: Logical volume "osd-block-054454e1-e7d0-4832-8347-50914f2b4b02" successfully removed /bin/podman: stderr: Removing physical volume "/dev/vdb" from volume group "ceph-94a5d8e2-7bbd-4a87-b6da-38fc7290a12c" /bin/podman: stdout: Volume group "ceph-94a5d8e2-7bbd-4a87-b6da-38fc7290a12c" successfully removed /bin/podman: Running command: /usr/bin/dd if=/dev/zero of=/dev/vdb bs=1M count=10 conv=fsync /bin/podman: --> Zapping successful for: <Raw Device: /dev/vdb> [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch device ls Hostname Path Type Serial Size Health Ident Fault Available ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash /dev/vdb hdd e1753d7b-cc8b-408a-9 16.1G Unknown N/A N/A Yes ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash /dev/vdc hdd 804acaa8-a5ff-4af2-8 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash /dev/vdd hdd 3560e149-2138-414a-b 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash /dev/vde hdd 842037ad-6533-4978-9 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash /dev/vdb hdd e4f0f5b9-888e-40ba-b 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash /dev/vdc hdd cc17dfbf-b177-46db-9 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash /dev/vdd hdd 7f12e9dc-bb16-48db-9 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash /dev/vde hdd 06de0da0-7fe2-4892-8 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash /dev/vdb hdd 61bbf7df-e0f4-4dd9-8 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash /dev/vdc hdd 54f8bc06-a6d5-4979-b 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash /dev/vdd hdd 4e463e20-2240-4364-9 16.1G Unknown N/A N/A No ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash /dev/vde hdd 6841af0b-a115-4360-b 16.1G Unknown N/A N/A No [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.16046 root default -7 0.04376 host ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash 5 hdd 0.01459 osd.5 up 1.00000 1.00000 8 hdd 0.01459 osd.8 up 1.00000 1.00000 11 hdd 0.01459 osd.11 up 1.00000 1.00000 -3 0.05835 host ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash 0 hdd 0.01459 osd.0 up 1.00000 1.00000 3 hdd 0.01459 osd.3 up 1.00000 1.00000 6 hdd 0.01459 osd.6 up 1.00000 1.00000 9 hdd 0.01459 osd.9 up 1.00000 1.00000 -5 0.05835 host ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash 1 hdd 0.01459 osd.1 up 1.00000 1.00000 4 hdd 0.01459 osd.4 up 1.00000 1.00000 7 hdd 0.01459 osd.7 up 1.00000 1.00000 10 hdd 0.01459 osd.10 up 1.00000 1.00000 [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch osd rm 5 --replace Scheduled OSD(s) for removal [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch osd rm status No OSD remove/replace operations reported [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.16046 root default -7 0.04376 host ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash 5 hdd 0.01459 osd.5 destroyed 0 1.00000 8 hdd 0.01459 osd.8 up 1.00000 1.00000 11 hdd 0.01459 osd.11 up 1.00000 1.00000 -3 0.05835 host ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash 0 hdd 0.01459 osd.0 up 1.00000 1.00000 3 hdd 0.01459 osd.3 up 1.00000 1.00000 6 hdd 0.01459 osd.6 up 1.00000 1.00000 9 hdd 0.01459 osd.9 up 1.00000 1.00000 -5 0.05835 host ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash 1 hdd 0.01459 osd.1 up 1.00000 1.00000 4 hdd 0.01459 osd.4 up 1.00000 1.00000 7 hdd 0.01459 osd.7 up 1.00000 1.00000 10 hdd 0.01459 osd.10 up 1.00000 1.00000 [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph orch daemon add osd ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash:/dev/vdb Created osd(s) 5 on host 'ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash' [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.16046 root default -7 0.04376 host ceph-sunil1adm-1614692246522-node3-osd-node-exporter-crash 5 hdd 0.01459 osd.5 up 1.00000 1.00000 8 hdd 0.01459 osd.8 up 1.00000 1.00000 11 hdd 0.01459 osd.11 up 1.00000 1.00000 -3 0.05835 host ceph-sunil1adm-1614692246522-node4-osd-node-exporter-crash 0 hdd 0.01459 osd.0 up 1.00000 1.00000 3 hdd 0.01459 osd.3 up 1.00000 1.00000 6 hdd 0.01459 osd.6 up 1.00000 1.00000 9 hdd 0.01459 osd.9 up 1.00000 1.00000 -5 0.05835 host ceph-sunil1adm-1614692246522-node5-osd-node-exporter-crash 1 hdd 0.01459 osd.1 up 1.00000 1.00000 4 hdd 0.01459 osd.4 up 1.00000 1.00000 7 hdd 0.01459 osd.7 up 1.00000 1.00000 10 hdd 0.01459 osd.10 up 1.00000 1.00000 [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# [ceph: root@ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor /]# ceph status cluster: id: f64f341c-655d-11eb-8778-fa163e914bcc health: HEALTH_WARN too many PGs per OSD (270 > max 250) services: mon: 3 daemons, quorum ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor,ceph-sunil1adm-1614692246522-node2-mon-mds-node-exporter-alertm,ceph-sunil1adm-1614692246522-node6-mon-rgw-node-exporter-crash (age 18h) mgr: ceph-sunil1adm-1614692246522-node1-mon-mgr-installer-node-expor.xpcyns(active, since 18h) mds: cephfs:1 {0=cephfs.ceph-sunil1adm-1614692246522-node8-client-nfs-node-exporter-isc.wzpisk=up:active} 1 up:standby osd: 11 osds: 11 up (since 13m), 11 in (since 13m) data: pools: 7 pools, 992 pgs objects: 172 objects, 24 KiB usage: 248 MiB used, 165 GiB / 165 GiB avail pgs: 992 active+clean io: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr
This is working as per the steps provided.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat Ceph Storage 5.0 bug fix and enhancement), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3294