Description of problem: When you set up OSDs with lvm scenario and set up dm-cache on top of them, the ceph-volume fails to run on top of such devices during playbook run or when executed locally as well: Output for dm-cache OSD [root@osds-0 ~]# ceph-volume lvm batch --report /dev/vdd --> RuntimeError: Unable to use device, already a member of LVM: /dev/vdd Output for NON dm-cache OSD - normal lvm [root@osds-1 ~]# ceph-volume lvm batch --report /dev/vdd --> All devices are already used by ceph. No OSDs will be created. Version-Release number of selected component (if applicable): ceph version 14.2.8-91.el8cp How reproducible: Always Steps to Reproduce: 1. Deploy lvm scenario 2. create dm-cache on top of OSDs 3. run "ceph-volume lvm batch --report <disk>" Actual results: fail Expected results: same behaviour as without dm-cache Additional info:
Cluster deployed with: - to force vdb, vdc act like SSD (as this is virtual machine): # ansible osds -m shell -a ' echo 0 > /sys/block/vdb/queue/rotational ' # ansible osds -m shell -a ' echo 0 > /sys/block/vdc/queue/rotational ' - deploy cluster # ansible-playbook site-container.yml # cat group_vars/osds.yml osd_scenario: lvm copy_admin_key: true devices: - /dev/vdc - /dev/vdd - /dev/vde ----- # ansible osds -m shell -a 'sgdisk --new=0:0:+14G -- /dev/vdb; sgdisk --new=0:0:+10G -- /dev/vdb; pvcreate /dev/vdb1; pvcreate /dev/vdb2' [root@osds-0 ~]# pvs; vgs; lvs PV VG Fmt Attr PSize PFree /dev/vdb1 lvm2 --- 14.00g 14.00g /dev/vdb2 lvm2 --- 10.00g 10.00g /dev/vdc ceph-block-dbs-f1ad666e-7db9-4124-a25f-ace6f190b68a lvm2 a-- <30.00g 4.00m /dev/vdd ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa lvm2 a-- <30.00g 0 /dev/vde ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171 lvm2 a-- <30.00g 0 VG #PV #LV #SN Attr VSize VFree ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171 1 1 0 wz--n- <30.00g 0 ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa 1 1 0 wz--n- <30.00g 0 ceph-block-dbs-f1ad666e-7db9-4124-a25f-ace6f190b68a 1 2 0 wz--n- <30.00g 4.00m LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert osd-block-de5e67e8-b562-42c9-b137-c0e28b0659f2 ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171 -wi-ao---- <30.00g osd-block-9b97f428-992d-420a-bc56-df26204d974e ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa -wi-ao---- <30.00g osd-block-db-05fdfa57-8d60-4000-bcc8-9f5c83ece9ec ceph-block-dbs-f1ad666e-7db9-4124-a25f-ace6f190b68a -wi-ao---- <15.00g osd-block-db-24857e40-d02d-42f0-83f1-f3ebc3544302 ceph-block-dbs-f1ad666e-7db9-4124-a25f-ace6f190b68a -wi-ao---- <15.00g [root@osds-0 ~]# systemctl stop ceph-osd@1 [root@osds-0 ~]# systemctl stop ceph-osd@4 [root@osds-0 ~]# lvchange -an ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171/^C [root@osds-0 ~]# vgextend ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171 /dev/vdb1 Volume group "ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171" successfully extended [root@osds-0 ~]# vgextend ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa /dev/vdb2 Volume group "ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa" successfully extended [root@osds-0 ~]# lvcreate -L 9G -n cachedisk ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171 /dev/vdb1 Logical volume "cachedisk" created. [root@osds-0 ~]# lvcreate -L 9G -n cachedisk ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa /dev/vdb2 Logical volume "cachedisk" created. [root@osds-0 ~]# lvchange -an ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171/cachedisk ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa/cachedisk ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171/osd-block-de5e67e8-b562-42c9-b137-c0e28b0659f2 ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa/osd-block-9b97f428-992d-420a-bc56-df26204d974e [root@osds-0 ~]# lvconvert --type writecache --cachevol cachedisk ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171/osd-block-de5e67e8-b562-42c9-b137-c0e28b0659f2 Erase all existing data on ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171/cachedisk? [y/n]: y Logical volume ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171/osd-block-de5e67e8-b562-42c9-b137-c0e28b0659f2 now has write cache. [root@osds-0 ~]# lvconvert --type writecache --cachevol cachedisk ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa/osd-block-9b97f428-992d-420a-bc56-df26204d974e Erase all existing data on ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa/cachedisk? [y/n]: y Logical volume ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa/osd-block-9b97f428-992d-420a-bc56-df26204d974e now has write cache. [root@osds-0 ~]# lvchange -ay ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171/cachedisk ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa/cachedisk ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171/osd-block-de5e67e8-b562-42c9-b137-c0e28b0659f2 ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa/osd-block-9b97f428-992d-420a-bc56-df26204d974e Failed to find logical volume "ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171/cachedisk" Failed to find logical volume "ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa/cachedisk" [root@osds-0 ~]# systemctl start ceph-osd@1 [root@osds-0 ~]# systemctl start ceph-osd@4 [root@osds-0 ~]# pvs; vgs; lvs PV VG Fmt Attr PSize PFree /dev/vdb1 ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171 lvm2 a-- <14.00g <5.00g /dev/vdb2 ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa lvm2 a-- <10.00g 1020.00m /dev/vdc ceph-block-dbs-f1ad666e-7db9-4124-a25f-ace6f190b68a lvm2 a-- <30.00g 4.00m /dev/vdd ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa lvm2 a-- <30.00g 0 /dev/vde ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171 lvm2 a-- <30.00g 0 VG #PV #LV #SN Attr VSize VFree ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171 2 1 0 wz--n- 43.99g <5.00g ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa 2 1 0 wz--n- 39.99g 1020.00m ceph-block-dbs-f1ad666e-7db9-4124-a25f-ace6f190b68a 1 2 0 wz--n- <30.00g 4.00m LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert osd-block-de5e67e8-b562-42c9-b137-c0e28b0659f2 ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171 Cwi-aoC--- <30.00g [cachedisk_cvol] [osd-block-de5e67e8-b562-42c9-b137-c0e28b0659f2_wcorig] 0.01 osd-block-9b97f428-992d-420a-bc56-df26204d974e ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa Cwi-aoC--- <30.00g [cachedisk_cvol] [osd-block-9b97f428-992d-420a-bc56-df26204d974e_wcorig] 0.01 osd-block-db-05fdfa57-8d60-4000-bcc8-9f5c83ece9ec ceph-block-dbs-f1ad666e-7db9-4124-a25f-ace6f190b68a -wi-ao---- <15.00g osd-block-db-24857e40-d02d-42f0-83f1-f3ebc3544302 ceph-block-dbs-f1ad666e-7db9-4124-a25f-ace6f190b68a -wi-ao---- <15.00g [root@osds-0 ~]# ceph-volume lvm batch --report /dev/vdd --> RuntimeError: Unable to use device, already a member of LVM: /dev/vdd
When rerunning "ansible-playbook site-container.yml" playbook fails on following TASK for node where dm-cache is set up TASK [ceph-config : run 'ceph-volume lvm batch --report' to see how many osds are to be created] ********************************************************************************************************************************************* task path: /usr/share/ceph-ansible/roles/ceph-config/tasks/main.yml:40 Tuesday 08 September 2020 05:32:50 -0400 (0:00:00.223) 0:05:37.147 ***** [WARNING]: The value 5120 (type int) in a string field was converted to '5120' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change. [WARNING]: The value -1 (type int) in a string field was converted to '-1' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change. changed: [10.10.95.125] => changed=true cmd: - podman - run - --rm - --privileged - --net=host - --ipc=host - --ulimit - nofile=1024:4096 - -v - /run/lock/lvm:/run/lock/lvm:z - -v - /var/run/udev/:/var/run/udev/:z - -v - /dev:/dev - -v - /etc/ceph:/etc/ceph:z - -v - /run/lvm/:/run/lvm/ - -v - /var/lib/ceph/:/var/lib/ceph/:z - -v - /var/log/ceph/:/var/log/ceph/:z - --entrypoint=ceph-volume - registry.redhat.io/rhceph/rhceph-4-rhel8:4-32 - --cluster - ceph - lvm - batch - --bluestore - --yes - --prepare - /dev/vdc - /dev/vdd - /dev/vde - --report - --format=json delta: '0:00:01.686883' end: '2020-09-08 05:32:52.626293' rc: 0 start: '2020-09-08 05:32:50.939410' stderr: |- WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. stderr_lines: <omitted> stdout: |- { "changed": false, "osds": [], "vgs": [] } stdout_lines: <omitted> fatal: [10.10.93.192]: FAILED! => changed=true cmd: - podman - run - --rm - --privileged - --net=host - --ipc=host - --ulimit - nofile=1024:4096 - -v - /run/lock/lvm:/run/lock/lvm:z - -v - /var/run/udev/:/var/run/udev/:z - -v - /dev:/dev - -v - /etc/ceph:/etc/ceph:z - -v - /run/lvm/:/run/lvm/ - -v - /var/lib/ceph/:/var/lib/ceph/:z - -v - /var/log/ceph/:/var/log/ceph/:z - --entrypoint=ceph-volume - registry.redhat.io/rhceph/rhceph-4-rhel8:4-32 - --cluster - ceph - lvm - batch - --bluestore - --yes - --prepare - /dev/vdc - /dev/vdd - /dev/vde - --report - --format=json msg: non-zero return code rc: 1 stderr: |- WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. --> RuntimeError: Unable to use device, already a member of LVM: /dev/vdd stderr_lines: <omitted> stdout: '' stdout_lines: <omitted> changed: [10.10.92.15] => changed=true cmd: - podman - run - --rm - --privileged - --net=host - --ipc=host - --ulimit - nofile=1024:4096 - -v - /run/lock/lvm:/run/lock/lvm:z - -v - /var/run/udev/:/var/run/udev/:z - -v - /dev:/dev - -v - /etc/ceph:/etc/ceph:z - -v - /run/lvm/:/run/lvm/ - -v - /var/lib/ceph/:/var/lib/ceph/:z - -v - /var/log/ceph/:/var/log/ceph/:z - --entrypoint=ceph-volume - registry.redhat.io/rhceph/rhceph-4-rhel8:4-32 - --cluster - ceph - lvm - batch - --bluestore - --yes - --prepare - /dev/vdc - /dev/vdd - /dev/vde - --report - --format=json delta: '0:00:02.095281' end: '2020-09-08 05:32:53.130394' rc: 0 start: '2020-09-08 05:32:51.035113' stderr: |- WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. stderr_lines: <omitted> stdout: |- { "changed": false, "osds": [], "vgs": [] } stdout_lines: <omitted>
tested with same results for combination of: RHEL 8.2 (4.18.0-193.el8.x86_64) + RHCS 4.1z1a2 (14.2.8-91.el8cp) [root@osds-0 ~]# ceph-volume lvm batch --report /dev/vdd --> RuntimeError: Unable to use device, already a member of LVM: /dev/vdd RHEL 7.8 (3.10.0-1127.el7.x86_64) + RHCS 3.3z6 (12.2.12-124.el7cp) [root@osds-2 ~]# ceph-volume lvm batch --report /dev/vdd --> RuntimeError: Unable to use device, already a member of LVM: /dev/vdd
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Ceph Storage security, bug fix, and enhancement Update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:1452