Bug 1876827

Summary:	[cee/sd][ceph-volume] "ceph-volume lvm batch --report /dev/vdd " fails on lvm OSDs set up with dm-cache
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Tomas Petr <tpetr>
Component:	Ceph-Volume	Assignee:	Andrew Schoen <aschoen>
Status:	CLOSED ERRATA	QA Contact:	Ameena Suhani S H <amsyedha>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.1	CC:	anharris, aschoen, ceph-eng-bugs, ceph-qe-bugs, gmeno, gsitlani, knortema, mhackett, mmuench, tserlin
Target Milestone:	---
Target Release:	4.2z1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-14.2.11-98.el8cp, ceph-14.2.11-98.el7cp	Doc Type:	Deprecated Functionality
Doc Text:	.Deploying an OSD on a device setup for `dm-cache` is unsupported Deploying an OSD with `dm-cache` is no longer supported. Use the Bluestore backend instead of `dm-cache`. For more information, see the link:{install-guide}#how-to-migrate-the-object-store-from-filestore-to-bluestore[_How to migrate the object store from FileStore to BlueStore_] section in the _{storage-product} Installation Guide_.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-04-28 20:12:31 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1890121

Description Tomas Petr 2020-09-08 09:34:35 UTC

Description of problem:
When you set up OSDs with lvm scenario and set up dm-cache  on top of them, the ceph-volume fails to run on top of such devices during playbook run or when executed locally as well:

Output for dm-cache OSD
[root@osds-0 ~]# ceph-volume lvm batch --report /dev/vdd
-->  RuntimeError: Unable to use device, already a member of LVM: /dev/vdd

Output for NON dm-cache OSD - normal lvm
[root@osds-1 ~]# ceph-volume lvm batch --report /dev/vdd
--> All devices are already used by ceph. No OSDs will be created.



Version-Release number of selected component (if applicable):
ceph version 14.2.8-91.el8cp


How reproducible:
Always

Steps to Reproduce:
1. Deploy lvm scenario
2. create dm-cache on top of OSDs
3. run "ceph-volume lvm batch --report <disk>"

Actual results:
fail

Expected results:
same behaviour as without dm-cache

Additional info:

Comment 1 Tomas Petr 2020-09-08 09:39:44 UTC

Cluster deployed with:

- to force vdb, vdc act like SSD (as this is virtual machine):
# ansible osds -m shell -a ' echo 0 > /sys/block/vdb/queue/rotational '
# ansible osds -m shell -a ' echo 0 > /sys/block/vdc/queue/rotational '
 - deploy cluster
# ansible-playbook site-container.yml
# cat group_vars/osds.yml
osd_scenario: lvm
copy_admin_key: true

devices:
  - /dev/vdc
  - /dev/vdd
  - /dev/vde

-----
#  ansible osds -m shell -a 'sgdisk --new=0:0:+14G -- /dev/vdb; sgdisk --new=0:0:+10G -- /dev/vdb; pvcreate /dev/vdb1; pvcreate /dev/vdb2'

[root@osds-0 ~]# pvs; vgs; lvs
  PV         VG                                                  Fmt  Attr PSize   PFree 
  /dev/vdb1                                                      lvm2 ---   14.00g 14.00g
  /dev/vdb2                                                      lvm2 ---   10.00g 10.00g
  /dev/vdc   ceph-block-dbs-f1ad666e-7db9-4124-a25f-ace6f190b68a lvm2 a--  <30.00g  4.00m
  /dev/vdd   ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa     lvm2 a--  <30.00g     0 
  /dev/vde   ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171     lvm2 a--  <30.00g     0 
  VG                                                  #PV #LV #SN Attr   VSize   VFree
  ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171       1   1   0 wz--n- <30.00g    0 
  ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa       1   1   0 wz--n- <30.00g    0 
  ceph-block-dbs-f1ad666e-7db9-4124-a25f-ace6f190b68a   1   2   0 wz--n- <30.00g 4.00m
  LV                                                VG                                                  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  osd-block-de5e67e8-b562-42c9-b137-c0e28b0659f2    ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171     -wi-ao---- <30.00g                                                    
  osd-block-9b97f428-992d-420a-bc56-df26204d974e    ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa     -wi-ao---- <30.00g                                                    
  osd-block-db-05fdfa57-8d60-4000-bcc8-9f5c83ece9ec ceph-block-dbs-f1ad666e-7db9-4124-a25f-ace6f190b68a -wi-ao---- <15.00g                                                    
  osd-block-db-24857e40-d02d-42f0-83f1-f3ebc3544302 ceph-block-dbs-f1ad666e-7db9-4124-a25f-ace6f190b68a -wi-ao---- <15.00g                                                    
[root@osds-0 ~]# systemctl stop ceph-osd@1
[root@osds-0 ~]# systemctl stop ceph-osd@4
[root@osds-0 ~]# lvchange -an ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171/^C
[root@osds-0 ~]# vgextend ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171 /dev/vdb1
  Volume group "ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171" successfully extended
[root@osds-0 ~]# vgextend ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa /dev/vdb2
  Volume group "ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa" successfully extended
[root@osds-0 ~]# lvcreate -L 9G -n cachedisk ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171 /dev/vdb1
  Logical volume "cachedisk" created.
[root@osds-0 ~]#  lvcreate -L 9G -n cachedisk ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa /dev/vdb2
  Logical volume "cachedisk" created.
[root@osds-0 ~]# lvchange -an ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171/cachedisk ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa/cachedisk ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171/osd-block-de5e67e8-b562-42c9-b137-c0e28b0659f2 ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa/osd-block-9b97f428-992d-420a-bc56-df26204d974e
[root@osds-0 ~]#  lvconvert --type writecache --cachevol cachedisk ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171/osd-block-de5e67e8-b562-42c9-b137-c0e28b0659f2
Erase all existing data on ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171/cachedisk? [y/n]: y
  Logical volume ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171/osd-block-de5e67e8-b562-42c9-b137-c0e28b0659f2 now has write cache.
[root@osds-0 ~]#  lvconvert --type writecache --cachevol cachedisk ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa/osd-block-9b97f428-992d-420a-bc56-df26204d974e
Erase all existing data on ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa/cachedisk? [y/n]: y
  Logical volume ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa/osd-block-9b97f428-992d-420a-bc56-df26204d974e now has write cache.
[root@osds-0 ~]# lvchange -ay ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171/cachedisk ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa/cachedisk ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171/osd-block-de5e67e8-b562-42c9-b137-c0e28b0659f2 ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa/osd-block-9b97f428-992d-420a-bc56-df26204d974e
  Failed to find logical volume "ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171/cachedisk"
  Failed to find logical volume "ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa/cachedisk"
[root@osds-0 ~]# systemctl start ceph-osd@1
[root@osds-0 ~]# systemctl start ceph-osd@4
[root@osds-0 ~]#  pvs; vgs; lvs
  PV         VG                                                  Fmt  Attr PSize   PFree   
  /dev/vdb1  ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171     lvm2 a--  <14.00g   <5.00g
  /dev/vdb2  ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa     lvm2 a--  <10.00g 1020.00m
  /dev/vdc   ceph-block-dbs-f1ad666e-7db9-4124-a25f-ace6f190b68a lvm2 a--  <30.00g    4.00m
  /dev/vdd   ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa     lvm2 a--  <30.00g       0 
  /dev/vde   ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171     lvm2 a--  <30.00g       0 
  VG                                                  #PV #LV #SN Attr   VSize   VFree   
  ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171       2   1   0 wz--n-  43.99g   <5.00g
  ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa       2   1   0 wz--n-  39.99g 1020.00m
  ceph-block-dbs-f1ad666e-7db9-4124-a25f-ace6f190b68a   1   2   0 wz--n- <30.00g    4.00m
  LV                                                VG                                                  Attr       LSize   Pool             Origin                                                  Data%  Meta%  Move Log Cpy%Sync Convert
  osd-block-de5e67e8-b562-42c9-b137-c0e28b0659f2    ceph-block-8fe3deaa-6b03-45c9-975e-12d5107ab171     Cwi-aoC--- <30.00g [cachedisk_cvol] [osd-block-de5e67e8-b562-42c9-b137-c0e28b0659f2_wcorig] 0.01                                   
  osd-block-9b97f428-992d-420a-bc56-df26204d974e    ceph-block-a4b04d43-666d-4be6-b182-3adba73757fa     Cwi-aoC--- <30.00g [cachedisk_cvol] [osd-block-9b97f428-992d-420a-bc56-df26204d974e_wcorig] 0.01                                   
  osd-block-db-05fdfa57-8d60-4000-bcc8-9f5c83ece9ec ceph-block-dbs-f1ad666e-7db9-4124-a25f-ace6f190b68a -wi-ao---- <15.00g                                                                                                                 
  osd-block-db-24857e40-d02d-42f0-83f1-f3ebc3544302 ceph-block-dbs-f1ad666e-7db9-4124-a25f-ace6f190b68a -wi-ao---- <15.00g 

[root@osds-0 ~]# ceph-volume lvm batch --report /dev/vdd
-->  RuntimeError: Unable to use device, already a member of LVM: /dev/vdd

Comment 2 Tomas Petr 2020-09-08 09:40:54 UTC

When rerunning "ansible-playbook site-container.yml" playbook fails on following TASK for node where dm-cache is set up

TASK [ceph-config : run 'ceph-volume lvm batch --report' to see how many osds are to be created] *********************************************************************************************************************************************
task path: /usr/share/ceph-ansible/roles/ceph-config/tasks/main.yml:40
Tuesday 08 September 2020  05:32:50 -0400 (0:00:00.223)       0:05:37.147 ***** 
[WARNING]: The value 5120 (type int) in a string field was converted to '5120' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change.

[WARNING]: The value -1 (type int) in a string field was converted to '-1' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change.

changed: [10.10.95.125] => changed=true 
  cmd:
  - podman
  - run
  - --rm
  - --privileged
  - --net=host
  - --ipc=host
  - --ulimit
  - nofile=1024:4096
  - -v
  - /run/lock/lvm:/run/lock/lvm:z
  - -v
  - /var/run/udev/:/var/run/udev/:z
  - -v
  - /dev:/dev
  - -v
  - /etc/ceph:/etc/ceph:z
  - -v
  - /run/lvm/:/run/lvm/
  - -v
  - /var/lib/ceph/:/var/lib/ceph/:z
  - -v
  - /var/log/ceph/:/var/log/ceph/:z
  - --entrypoint=ceph-volume
  - registry.redhat.io/rhceph/rhceph-4-rhel8:4-32
  - --cluster
  - ceph
  - lvm
  - batch
  - --bluestore
  - --yes
  - --prepare
  - /dev/vdc
  - /dev/vdd
  - /dev/vde
  - --report
  - --format=json
  delta: '0:00:01.686883'
  end: '2020-09-08 05:32:52.626293'
  rc: 0
  start: '2020-09-08 05:32:50.939410'
  stderr: |-
    WARNING: The same type, major and minor should not be used for multiple devices.
    WARNING: The same type, major and minor should not be used for multiple devices.
    WARNING: The same type, major and minor should not be used for multiple devices.
    WARNING: The same type, major and minor should not be used for multiple devices.
  stderr_lines: <omitted>
  stdout: |-
    {
        "changed": false,
        "osds": [],
        "vgs": []
    }
  stdout_lines: <omitted>
fatal: [10.10.93.192]: FAILED! => changed=true 
  cmd:
  - podman
  - run
  - --rm
  - --privileged
  - --net=host
  - --ipc=host
  - --ulimit
  - nofile=1024:4096
  - -v
  - /run/lock/lvm:/run/lock/lvm:z
  - -v
  - /var/run/udev/:/var/run/udev/:z
  - -v
  - /dev:/dev
  - -v
  - /etc/ceph:/etc/ceph:z
  - -v
  - /run/lvm/:/run/lvm/
  - -v
  - /var/lib/ceph/:/var/lib/ceph/:z
  - -v
  - /var/log/ceph/:/var/log/ceph/:z
  - --entrypoint=ceph-volume
  - registry.redhat.io/rhceph/rhceph-4-rhel8:4-32
  - --cluster
  - ceph
  - lvm
  - batch
  - --bluestore
  - --yes
  - --prepare
  - /dev/vdc
  - /dev/vdd
  - /dev/vde
  - --report
  - --format=json
  msg: non-zero return code
  rc: 1
  stderr: |-
    WARNING: The same type, major and minor should not be used for multiple devices.
    WARNING: The same type, major and minor should not be used for multiple devices.
    WARNING: The same type, major and minor should not be used for multiple devices.
    WARNING: The same type, major and minor should not be used for multiple devices.
    -->  RuntimeError: Unable to use device, already a member of LVM: /dev/vdd
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>
changed: [10.10.92.15] => changed=true 
  cmd:
  - podman
  - run
  - --rm
  - --privileged
  - --net=host
  - --ipc=host
  - --ulimit
  - nofile=1024:4096
  - -v
  - /run/lock/lvm:/run/lock/lvm:z
  - -v
  - /var/run/udev/:/var/run/udev/:z
  - -v
  - /dev:/dev
  - -v
  - /etc/ceph:/etc/ceph:z
  - -v
  - /run/lvm/:/run/lvm/
  - -v
  - /var/lib/ceph/:/var/lib/ceph/:z
  - -v
  - /var/log/ceph/:/var/log/ceph/:z
  - --entrypoint=ceph-volume
  - registry.redhat.io/rhceph/rhceph-4-rhel8:4-32
  - --cluster
  - ceph
  - lvm
  - batch
  - --bluestore
  - --yes
  - --prepare
  - /dev/vdc
  - /dev/vdd
  - /dev/vde
  - --report
  - --format=json
  delta: '0:00:02.095281'
  end: '2020-09-08 05:32:53.130394'
  rc: 0
  start: '2020-09-08 05:32:51.035113'
  stderr: |-
    WARNING: The same type, major and minor should not be used for multiple devices.
    WARNING: The same type, major and minor should not be used for multiple devices.
    WARNING: The same type, major and minor should not be used for multiple devices.
    WARNING: The same type, major and minor should not be used for multiple devices.
  stderr_lines: <omitted>
  stdout: |-
    {
        "changed": false,
        "osds": [],
        "vgs": []
    }
  stdout_lines: <omitted>

Comment 3 Tomas Petr 2020-09-09 15:50:38 UTC

tested with same results for combination of:

RHEL 8.2 (4.18.0-193.el8.x86_64) + RHCS 4.1z1a2 (14.2.8-91.el8cp)
[root@osds-0 ~]# ceph-volume lvm batch --report /dev/vdd
-->  RuntimeError: Unable to use device, already a member of LVM: /dev/vdd

RHEL 7.8 (3.10.0-1127.el7.x86_64) + RHCS 3.3z6 (12.2.12-124.el7cp)
[root@osds-2 ~]#  ceph-volume lvm batch --report /dev/vdd
-->  RuntimeError: Unable to use device, already a member of LVM: /dev/vdd

Comment 17 errata-xmlrpc 2021-04-28 20:12:31 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage security, bug fix, and enhancement Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1452