Description of problem: After a normal PV is remove and the underling storage is removed, it properly doesn't cause a warning. PVs on stacked LVs do. [root@hayes-01 ~]# pvcreate /dev/sdg1 Physical volume "/dev/sdg1" successfully created. [root@hayes-01 ~]# vgcreate vg /dev/sdg1 Volume group "vg" successfully created with system ID hayes-01.lab.msp.redhat.com [root@hayes-01 ~]# lvcreate --type linear -n lv_stack -L 500M vg Logical volume "lv_stack" created. [root@hayes-01 ~]# pvscan PV /dev/sdg1 VG vg lvm2 [<1.82 TiB / <1.82 TiB free] Total: 1 [<1.82 TiB] / in use: 1 [<1.82 TiB] / in no VG: 0 [0 ] [root@hayes-01 ~]# pvcreate --config devices/scan_lvs=1 /dev/vg/lv_stack Physical volume "/dev/vg/lv_stack" successfully created. [root@hayes-01 ~]# grep stack /etc/lvm/devices/system.devices IDTYPE=lvmlv_uuid IDNAME=LVM-DsL7OSnInKDkkVvkKkFGabtdfK9SevEPLH0eWngN1onjDaONci8g7hfQW7j62Fq1 DEVNAME=/dev/vg/lv_stack PVID=VAxdEeDwB80Ul3Ge5j6KpYKzLI31uBZN [root@hayes-01 ~]# pvremove --config devices/scan_lvs=1 /dev/vg/lv_stack Labels on physical volume "/dev/vg/lv_stack" successfully wiped. # PVID is removed [root@hayes-01 ~]# grep stack /etc/lvm/devices/system.devices IDTYPE=lvmlv_uuid IDNAME=LVM-DsL7OSnInKDkkVvkKkFGabtdfK9SevEPLH0eWngN1onjDaONci8g7hfQW7j62Fq1 DEVNAME=/dev/vg/lv_stack PVID=. [root@hayes-01 ~]# pvscan PV /dev/sdg1 VG vg lvm2 [<1.82 TiB / <1.82 TiB free] Total: 1 [<1.82 TiB] / in use: 1 [<1.82 TiB] / in no VG: 0 [0 ] [root@hayes-01 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert lv_stack vg -wi-a----- 500.00m [root@hayes-01 ~]# vgremove -f vg Logical volume "lv_stack" successfully removed. Volume group "vg" successfully removed # lvmlv_uuid device missing warning now appears [root@hayes-01 ~]# pvscan Devices file lvmlv_uuid LVM-DsL7OSnInKDkkVvkKkFGabtdfK9SevEPLH0eWngN1onjDaONci8g7hfQW7j62Fq1 PVID none last seen on /dev/vg/lv_stack not found. PV /dev/sdg1 lvm2 [<1.82 TiB] Total: 1 [<1.82 TiB] / in use: 0 [0 ] / in no VG: 1 [<1.82 TiB] [root@hayes-01 ~]# grep stack /etc/lvm/devices/system.devices IDTYPE=lvmlv_uuid IDNAME=LVM-DsL7OSnInKDkkVvkKkFGabtdfK9SevEPLH0eWngN1onjDaONci8g7hfQW7j62Fq1 DEVNAME=/dev/vg/lv_stack PVID=. [root@hayes-01 ~]# pvremove -f /dev/sdg1 Devices file lvmlv_uuid LVM-DsL7OSnInKDkkVvkKkFGabtdfK9SevEPLH0eWngN1onjDaONci8g7hfQW7j62Fq1 PVID none last seen on /dev/vg/lv_stack not found. Labels on physical volume "/dev/sdg1" successfully wiped. # I removed the backing device for the last PV /dev/sdg1 as well, but there is no warning about that device not found [root@hayes-01 ~]# echo offline > /sys/block/sdg/device/state [root@hayes-01 ~]# pvscan Devices file lvmlv_uuid LVM-DsL7OSnInKDkkVvkKkFGabtdfK9SevEPLH0eWngN1onjDaONci8g7hfQW7j62Fq1 PVID none last seen on /dev/vg/lv_stack not found. No matching physical volumes found [root@hayes-01 ~]# cat /etc/lvm/devices/system.devices # LVM uses devices listed in this file. # Created by LVM command pvremove pid 50750 at Thu Jul 15 17:05:20 2021 SYSTEMID=hayes-01.lab.msp.redhat.com VERSION=1.1.391 IDTYPE=sys_wwid IDNAME=naa.6d094660575ece002291bade2acdbd28 DEVNAME=/dev/sde1 PVID=. PART=1 IDTYPE=sys_wwid IDNAME=naa.6d094660575ece002291bb4a313e4759 DEVNAME=/dev/sdf1 PVID=. PART=1 IDTYPE=sys_wwid IDNAME=naa.6d094660575ece002291be7e6227ca72 DEVNAME=/dev/sdi1 PVID=. PART=1 IDTYPE=sys_wwid IDNAME=naa.6d094660575ece002291bf456dfeb02b DEVNAME=/dev/sdj1 PVID=. PART=1 IDTYPE=sys_wwid IDNAME=naa.6d094660575ece002291ba5c230d16c6 DEVNAME=/dev/sdd1 PVID=. PART=1 IDTYPE=sys_wwid IDNAME=naa.6d094660575ece002291b9e01bad8691 DEVNAME=/dev/sdc1 PVID=. PART=1 IDTYPE=sys_wwid IDNAME=naa.6d094660575ece002291b958138a7c80 DEVNAME=/dev/sdb1 PVID=. PART=1 IDTYPE=sys_wwid IDNAME=naa.6d094660575ece002291bdea594f2502 DEVNAME=/dev/sdh1 PVID=. PART=1 IDTYPE=sys_wwid IDNAME=naa.6d094660575ece002291bd67517f677a DEVNAME=/dev/sdg1 PVID=. PART=1 IDTYPE=lvmlv_uuid IDNAME=LVM-DsL7OSnInKDkkVvkKkFGabtdfK9SevEPLH0eWngN1onjDaONci8g7hfQW7j62Fq1 DEVNAME=/dev/vg/lv_stack PVID=. Version-Release number of selected component (if applicable): kernel-4.18.0-319.el8 BUILT: Mon Jun 28 20:52:51 CDT 2021 lvm2-2.03.12-5.el8 BUILT: Tue Jul 13 11:50:03 CDT 2021 lvm2-libs-2.03.12-5.el8 BUILT: Tue Jul 13 11:50:03 CDT 2021
The general principle is that the user needs to explicitly add and remove devices from the devices file using lvmdevices --adddev|--deldev. So, pvremove doesn't remove the PV from the file, it just clears the PVID field as shown. As a way to ease the user transition, we made a special case for pvcreate/vgcreate so that those commands *do* add the target device to the devices file. This exception is technically a violation of the devices file concept, and we may at some point add a config setting to turn off this exception for pvcreate/vgcreate. In that case, a user would need to do lvmdevices --adddev prior to running pvcreate on the device. We've not come up with any exceptions where a command should automatically *remove* a device from the devices file, as we have for adding devices. It's possible that a case might come up which we've not thought of (possibly the stacking case if we find that's a recurring issue?) The stacking scenario where an LV is the device is being handled in the same way as a scsi device, where the user needs to explicitly remove it from the devices file if it's permanently detached from the system. (In the test above, setting the state to "offline" isn't actually detaching the device, it's still visible on the system even while offline.) [root@null-03 ~]# pvs PV VG Fmt Attr PSize PFree /dev/sda2 rhel_null-03 lvm2 a-- 465.27g 0 /dev/sdf ff lvm2 a-- 931.01g 931.01g [root@null-03 ~]# lvmdevices Device /dev/sdf IDTYPE=sys_wwid IDNAME=naa.690b11c0000438ad0000057150910423 DEVNAME=/dev/sdf PVID=8hkmvJmTBHhnzn63RFXEFAdEy64I24HM Device /dev/sdb IDTYPE=sys_wwid IDNAME=naa.690b11c0000438ad0000056550910404 DEVNAME=/dev/sdb PVID=b1BaIjgRKsoXI9CGdWhJ50kKPaOZDZzo Device /dev/sda2 IDTYPE=sys_wwid IDNAME=naa.50014ee0ae1598c2 DEVNAME=/dev/sda2 PVID=0GfvU7fNrko5iAn1UIVOoKN2JyZr1s8z PART=2 [root@null-03 ~]# pvs /dev/sdb Cannot use PV /dev/sdb in shared VG bbsan. [root@null-03 ~]# pvs /dev/sdf PV VG Fmt Attr PSize PFree /dev/sdf ff lvm2 a-- 931.01g 931.01g [root@null-03 ~]# echo "offline" > /sys/block/sdf/device/state [root@null-03 ~]# pvs PV VG Fmt Attr PSize PFree /dev/sda2 rhel_null-03 lvm2 a-- 465.27g 0 [root@null-03 ~]# pvs /dev/sdf Cannot use /dev/sdf: device is too small (pv_min_size) [root@null-03 ~]# cat /sys/block/sdf/device/state offline [root@null-03 ~]# cat /sys/block/sdf/device/wwid naa.690b11c0000438ad0000057150910423
AFAICT we can probably close this, but parking in need info for a bit in case there's some usability aspect to this that we should look at.
You're correct, this can be closed. This LV stack removal condition is no different than any other device being removed with an existing devicesfile entry. Plus, we also already have bug 1956966 to deal with the potential annoyance of stale device/partition entries. # A similar md example. [root@hayes-01 ~]# pvcreate /dev/md127 Physical volume "/dev/md127" successfully created. [root@hayes-01 ~]# grep md /etc/lvm/devices/system.devices IDTYPE=md_uuid IDNAME=24904443-4c7a-7e4f-2f97-8ecfec58aeea DEVNAME=/dev/md127 PVID=f1lfjzc78b3oqO9xfbvjCqInli84OHwT [root@hayes-01 ~]# pvremove /dev/md127 Labels on physical volume "/dev/md127" successfully wiped. [root@hayes-01 ~]# grep md /etc/lvm/devices/system.devices IDTYPE=md_uuid IDNAME=24904443-4c7a-7e4f-2f97-8ecfec58aeea DEVNAME=/dev/md127 PVID=. [root@hayes-01 ~]# mdadm --stop /dev/md127 mdadm: stopped /dev/md127 [root@hayes-01 ~]# mdadm --zero-superblock /dev/sdc1 /dev/sde1 [root@hayes-01 ~]# pvscan Devices file md_uuid 24904443-4c7a-7e4f-2f97-8ecfec58aeea PVID none last seen on /dev/md127 not found. No matching physical volumes found