Created attachment 1087306 [details] output from vgremove -f fubar -vvvv Description of problem: When you remove a VG that has a LV which is a PV itself, lvmetad maintains the removed PV reference and emits a warning for any command run after it. Version-Release number of selected component (if applicable): LVM version: 2.02.132(2) (2015-09-22) Library version: 1.02.109 (2015-09-22) Driver version: 4.33.0 How reproducible: Always Steps to Reproduce: [root@localhost tools]# pvs PV VG Fmt Attr PSize PFree /dev/sdb lvm2 --- 18.00g 18.00g /dev/sdc lvm2 --- 18.00g 18.00g /dev/sdd lvm2 --- 18.00g 18.00g /dev/sde lvm2 --- 18.00g 18.00g [root@localhost tools]# vgcreate fubar /dev/sdb /dev/sdc Volume group "fubar" successfully created [root@localhost tools]# lvcreate -L64m -n nested fubar Logical volume "nested" created. [root@localhost tools]# pvcreate /dev/fubar/nested Physical volume "/dev/fubar/nested" successfully created [root@localhost tools]# vgremove -f fubar Logical volume "nested" successfully removed Volume group "fubar" successfully removed [root@localhost tools]# pvs WARNING: Device for PV xRJsFD-hbjJ-hq3r-Tsa2-o718-4nii-j6a7GA not found or rejected by a filter. PV VG Fmt Attr PSize PFree /dev/sdb lvm2 --- 18.00g 18.00g /dev/sdc lvm2 --- 18.00g 18.00g /dev/sdd lvm2 --- 18.00g 18.00g /dev/sde lvm2 --- 18.00g 18.00g [root@localhost tools]# systemctl restart lvm2-lvmetad.service [root@localhost tools]# pvs PV VG Fmt Attr PSize PFree /dev/sdb lvm2 --- 18.00g 18.00g /dev/sdc lvm2 --- 18.00g 18.00g /dev/sdd lvm2 --- 18.00g 18.00g /dev/sde lvm2 --- 18.00g 18.00g [root@localhost tools]# Actual results: Warning is emitted for a missing PV device Expected results: No warning Additional info:
You don't need to remove the VG, you can just do vgchange -an fubar to deactivate the lv, making the device under the PV disappear. To rescan and update lvmetad you only need to run 'pvscan --cache' as opposed to restarting the daemon. Using LVs as PVs is not a fully automated feature and requires you to perform some steps manually that the system would usually perform. In particular, LVs as PVs is not integrated with udev. So, you won't get fully functioning activation of LVs layered like this, and you won't get PV/device events which is the issue you're seeing here. In theory, when you do vgchange -an to deactivate the LV which is a PV, you would get a "device removed" event based on the fact that the LV was acting as a device. It seems unlikely to me that we'd integrate this kind of layering into the current uevent scheme given the challenges we already have in that area.
(In reply to David Teigland from comment #1) > You don't need to remove the VG, you can just do vgchange -an fubar to > deactivate the lv, making the device under the PV disappear. I simply wanted to remove the VG and everything is contained. Seems to me that when you do the -f it should be equivalent to: # pvremove /dev/fubar/nested # lvremove fubar/nested # vgremove fubar without the confirmations. However the code doesn't appear to check that the LV is also a PV, so it skips the first step. If you run vgremove without the '-f' you get the same result, cache in wrong state. > To rescan and > update lvmetad you only need to run 'pvscan --cache' as opposed to > restarting the daemon. Requiring a user to correct the cache state at anytime seems very wrong to me. In fact if lvmetad wasn't running in this case, lvm would have worked just fine. > Using LVs as PVs is not a fully automated feature and requires you to > perform some steps manually that the system would usually perform. In > particular, LVs as PVs is not integrated with udev. Well I'm seeing udev events when I run the pvcreate on /dev/fubar/nested LV and when I create a LV in the VG which resides on the embedded VG, but maybe I'm missing something subtle. > So, you won't get fully > functioning activation of LVs layered like this, and you won't get PV/device > events which is the issue you're seeing here. In theory, when you do > vgchange -an to deactivate the LV which is a PV, you would get a "device > removed" event based on the fact that the LV was acting as a device. It > seems unlikely to me that we'd integrate this kind of layering into the > current uevent scheme given the challenges we already have in that area. So what you're saying is LVM nesting isn't really a supported feature?
I just meant that you can reproduce the problem more simply and directly by using vgchange -an. Layering lvm on lvm works well enough to be useful for some things (and I'd guess well enough for what most people need it for.)
Stacks should be working with lvmetad and autoactivation (the LV is just like any other disk/PV for which we listen on uevents to update lvmetad and trigger the autoactivation): [0] raw/~ # vgcreate vg1 /dev/sda Physical volume "/dev/sda" successfully created Volume group "vg1" successfully created [0] raw/~ # lvcreate -l100%FREE vg1 Logical volume "lvol0" created. [0] raw/~ # vgcreate vg2 /dev/vg1/lvol0 Physical volume "/dev/vg1/lvol0" successfully created Volume group "vg2" successfully created [0] raw/~ # lvcreate -l100%FREE vg2 Logical volume "lvol0" created. [0] raw/~ # lsblk /dev/sda NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 128M 0 disk `-vg1-lvol0 253:2 0 124M 0 lvm `-vg2-lvol0 253:3 0 120M 0 lvm `-vg3-lvol0 253:4 0 116M 0 lvm [0] raw/~ # vgchange -an vg1 Logical volume vg1/lvol0 is used by another device. Can't deactivate volume group "vg1" with 1 open logical volume(s) [0] raw/~ # vgchange -an vg3 0 logical volume(s) in volume group "vg3" now active [0] raw/~ # vgchange -an vg2 0 logical volume(s) in volume group "vg2" now active [0] raw/~ # vgchange -an vg1 0 logical volume(s) in volume group "vg1" now active [0] raw/~ # lsblk /dev/sda NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 128M 0 disk (now we activate only the bottom-level vg1/lvol0 and all the stack above is autoactivated because of uevents that trigger the pvscan --cache -aay in udev rules) [0] raw/~ # vgchange -ay vg1 1 logical volume(s) in volume group "vg1" now active [0] raw/~ # lsblk /dev/sda NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 128M 0 disk `-vg1-lvol0 253:2 0 124M 0 lvm `-vg2-lvol0 253:3 0 120M 0 lvm `-vg3-lvol0 253:4 0 116M 0 lvm
Yes, this doesn't work though (but that's a bug!): 0] raw/~ # vgcreate vg1 /dev/sda Physical volume "/dev/sda" successfully created Volume group "vg1" successfully created [0] raw/~ # lvcreate -l100%FREE vg1 Logical volume "lvol0" created. [0] raw/~ # vgcreate vg2 /dev/vg1/lvol0 Physical volume "/dev/vg1/lvol0" successfully created Volume group "vg2" successfully created [0] raw/~ # vgchange -an vg1 0 logical volume(s) in volume group "vg1" now active [0] raw/~ # pvs WARNING: Device for PV MjuKj0-jw4k-Cbtt-PrWO-3XCc-yqK5-530dGv not found or rejected by a filter. WARNING: Device for PV MjuKj0-jw4k-Cbtt-PrWO-3XCc-yqK5-530dGv not found or rejected by a filter. PV VG Fmt Attr PSize PFree /dev/sda vg1 lvm2 a-- 124.00m 0 /dev/vda2 fedora lvm2 a-- 19.49g 0 unknown device vg2 lvm2 a-m 120.00m 120.00m === However, it works after reboot: [0] raw/~ # vgs VG #PV #LV #SN Attr VSize VFree fedora 1 2 0 wz--n- 19.49g 0 vg1 1 1 0 wz--n- 124.00m 0 vg2 1 0 0 wz--n- 120.00m 120.00m [0] raw/~ # vgchange -an vg1 0 logical volume(s) in volume group "vg1" now active [0] raw/~ # pvs PV VG Fmt Attr PSize PFree /dev/sda vg1 lvm2 a-- 124.00m 0 /dev/vda2 fedora lvm2 a-- 19.49g 0 I'll have a look...
Ah yes, I know what's wrong! The issue here is that there's no lvm2-pvscan@major:minor.service for the LV that acts as a PV: [0] raw/~ # vgcreate vg1 /dev/sda Physical volume "/dev/sda" successfully created Volume group "vg1" successfully created [0] raw/~ # lvcreate -l100%FREE vg1 Logical volume "lvol0" created. [0] raw/~ # vgcreate vg2 /dev/vg1/lvol0 Physical volume "/dev/vg1/lvol0" successfully created Volume group "vg2" successfully created [0] raw/~ # lvs -o name,kernel_major,kernel_minor vg1/lvol0 LV KMaj KMin lvol0 253 2 [0] raw/~ # systemctl status lvm2-pvscan@253:2.service * lvm2-pvscan@253:2.service - LVM2 PV scan on device 253:2 Loaded: loaded (/usr/lib/systemd/system/lvm2-pvscan@.service; static; vendor preset: disabled) Active: inactive (dead) Docs: man:pvscan(8) (...reboot...) 0] raw/~ # systemctl status lvm2-pvscan@253:2.service * lvm2-pvscan@253:2.service - LVM2 PV scan on device 253:2 Loaded: loaded (/usr/lib/systemd/system/lvm2-pvscan@.service; static; vendor preset: disabled) Active: active (exited) since Fri 2015-10-30 13:10:04 CET; 11s ago Docs: man:pvscan(8) Process: 662 ExecStart=/usr/sbin/lvm pvscan --cache --activate ay %i (code=exited, status=0/SUCCESS) Main PID: 662 (code=exited, status=0/SUCCESS) Oct 30 13:10:04 raw.virt systemd[1]: Starting LVM2 PV scan on device 253:2... Oct 30 13:10:04 raw.virt lvm[662]: 0 logical volume(s) in volume group "vg2" now active Oct 30 13:10:04 raw.virt systemd[1]: Started LVM2 PV scan on device 253:2. This means, there's no hook to call pvscan --cache major:minor for the removed LV (that is a PV at the same time) - this is normally executed as the lvm2-pvscan@major:minor.service's ExecStop action. But since we don't have the service...
This bug appears to have been reported against 'rawhide' during the Fedora 24 development cycle. Changing version to '24'. More information and reason for this action is here: https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora24#Rawhide_Rebase
This message is a reminder that Fedora 24 is nearing its end of life. Approximately 2 (two) weeks from now Fedora will stop maintaining and issuing updates for Fedora 24. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '24'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 24 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
This message is a reminder that Fedora 26 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 26. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '26'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 26 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.