Bug 1276116 - lvmetad: Removing VG with nested PV causes warning to be emitted until service restarted
lvmetad: Removing VG with nested PV causes warning to be emitted until servic...
Status: NEW
Product: Fedora
Classification: Fedora
Component: lvm2 (Show other bugs)
26
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Peter Rajnoha
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-28 14:44 EDT by Tony Asleson
Modified: 2017-07-25 15:41 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
output from vgremove -f fubar -vvvv (57.04 KB, text/plain)
2015-10-28 14:44 EDT, Tony Asleson
no flags Details

  None (edit)
Description Tony Asleson 2015-10-28 14:44:28 EDT
Created attachment 1087306 [details]
output from vgremove -f fubar -vvvv

Description of problem:

When you remove a VG that has a LV which is a PV itself, lvmetad maintains the removed PV reference and emits a warning for any command run after it.

Version-Release number of selected component (if applicable):

LVM version:     2.02.132(2) (2015-09-22)
Library version: 1.02.109 (2015-09-22)
Driver version:  4.33.0

How reproducible:

Always

Steps to Reproduce:

[root@localhost tools]# pvs
  PV         VG   Fmt  Attr PSize  PFree 
  /dev/sdb        lvm2 ---  18.00g 18.00g
  /dev/sdc        lvm2 ---  18.00g 18.00g
  /dev/sdd        lvm2 ---  18.00g 18.00g
  /dev/sde        lvm2 ---  18.00g 18.00g
[root@localhost tools]# vgcreate fubar /dev/sdb /dev/sdc
  Volume group "fubar" successfully created
[root@localhost tools]# lvcreate -L64m -n nested fubar
  Logical volume "nested" created.
[root@localhost tools]# pvcreate /dev/fubar/nested
  Physical volume "/dev/fubar/nested" successfully created
[root@localhost tools]# vgremove -f fubar
  Logical volume "nested" successfully removed
  Volume group "fubar" successfully removed
[root@localhost tools]# pvs
  WARNING: Device for PV xRJsFD-hbjJ-hq3r-Tsa2-o718-4nii-j6a7GA not found or rejected by a filter.
  PV         VG   Fmt  Attr PSize  PFree 
  /dev/sdb        lvm2 ---  18.00g 18.00g
  /dev/sdc        lvm2 ---  18.00g 18.00g
  /dev/sdd        lvm2 ---  18.00g 18.00g
  /dev/sde        lvm2 ---  18.00g 18.00g
[root@localhost tools]# systemctl restart lvm2-lvmetad.service
[root@localhost tools]# pvs
  PV         VG   Fmt  Attr PSize  PFree 
  /dev/sdb        lvm2 ---  18.00g 18.00g
  /dev/sdc        lvm2 ---  18.00g 18.00g
  /dev/sdd        lvm2 ---  18.00g 18.00g
  /dev/sde        lvm2 ---  18.00g 18.00g
[root@localhost tools]# 

Actual results:
Warning is emitted for a missing PV device

Expected results:
No warning

Additional info:
Comment 1 David Teigland 2015-10-28 15:07:20 EDT
You don't need to remove the VG, you can just do vgchange -an fubar to deactivate the lv, making the device under the PV disappear.  To rescan and update lvmetad you only need to run 'pvscan --cache' as opposed to restarting the daemon.

Using LVs as PVs is not a fully automated feature and requires you to perform some steps manually that the system would usually perform.  In particular, LVs as PVs is not integrated with udev.  So, you won't get fully functioning activation of LVs layered like this, and you won't get PV/device events which is the issue you're seeing here.  In theory, when you do vgchange -an to deactivate the LV which is a PV, you would get a "device removed" event based on the fact that the LV was acting as a device.  It seems unlikely to me that we'd integrate this kind of layering into the current uevent scheme given the challenges we already have in that area.
Comment 2 Tony Asleson 2015-10-28 15:53:44 EDT
(In reply to David Teigland from comment #1)
> You don't need to remove the VG, you can just do vgchange -an fubar to
> deactivate the lv, making the device under the PV disappear.

I simply wanted to remove the VG and everything is contained.  Seems to me that when you do the -f it should be equivalent to:

# pvremove /dev/fubar/nested
# lvremove fubar/nested
# vgremove fubar

without the confirmations.  However the code doesn't appear to check that the LV is also a PV, so it skips the first step.  If you run vgremove without the '-f' you get the same result, cache in wrong state.

> To rescan and
> update lvmetad you only need to run 'pvscan --cache' as opposed to
> restarting the daemon.

Requiring a user to correct the cache state at anytime seems very wrong to me.  In fact if lvmetad wasn't running in this case, lvm would have worked just fine.

> Using LVs as PVs is not a fully automated feature and requires you to
> perform some steps manually that the system would usually perform.  In
> particular, LVs as PVs is not integrated with udev.  

Well I'm seeing udev events when I run the pvcreate on /dev/fubar/nested LV and when I create a LV in the VG which resides on the embedded VG, but maybe I'm missing something subtle.

> So, you won't get fully
> functioning activation of LVs layered like this, and you won't get PV/device
> events which is the issue you're seeing here.  In theory, when you do
> vgchange -an to deactivate the LV which is a PV, you would get a "device
> removed" event based on the fact that the LV was acting as a device.  It
> seems unlikely to me that we'd integrate this kind of layering into the
> current uevent scheme given the challenges we already have in that area.

So what you're saying is LVM nesting isn't really a supported feature?
Comment 3 David Teigland 2015-10-28 16:23:26 EDT
I just meant that you can reproduce the problem more simply and directly by using vgchange -an.

Layering lvm on lvm works well enough to be useful for some things (and I'd guess well enough for what most people need it for.)
Comment 4 Peter Rajnoha 2015-10-30 07:49:01 EDT
Stacks should be working with lvmetad and autoactivation (the LV is just like any other disk/PV for which we listen on uevents to update lvmetad and trigger the autoactivation):

[0] raw/~ # vgcreate vg1 /dev/sda
  Physical volume "/dev/sda" successfully created
  Volume group "vg1" successfully created

[0] raw/~ # lvcreate -l100%FREE vg1
  Logical volume "lvol0" created.

[0] raw/~ # vgcreate vg2 /dev/vg1/lvol0
  Physical volume "/dev/vg1/lvol0" successfully created
  Volume group "vg2" successfully created

[0] raw/~ # lvcreate -l100%FREE vg2
  Logical volume "lvol0" created.

[0] raw/~ # lsblk /dev/sda
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0  128M  0 disk 
`-vg1-lvol0     253:2    0  124M  0 lvm  
  `-vg2-lvol0   253:3    0  120M  0 lvm  
    `-vg3-lvol0 253:4    0  116M  0 lvm  

[0] raw/~ # vgchange -an vg1
  Logical volume vg1/lvol0 is used by another device.
  Can't deactivate volume group "vg1" with 1 open logical volume(s)

[0] raw/~ # vgchange -an vg3
  0 logical volume(s) in volume group "vg3" now active

[0] raw/~ # vgchange -an vg2
  0 logical volume(s) in volume group "vg2" now active

[0] raw/~ # vgchange -an vg1
  0 logical volume(s) in volume group "vg1" now active

[0] raw/~ # lsblk /dev/sda
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0  128M  0 disk 


(now we activate only the bottom-level vg1/lvol0 and all the stack above is autoactivated because of uevents that trigger the pvscan --cache -aay in udev rules)

[0] raw/~ # vgchange -ay vg1
  1 logical volume(s) in volume group "vg1" now active

[0] raw/~ # lsblk /dev/sda
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0  128M  0 disk 
`-vg1-lvol0     253:2    0  124M  0 lvm  
  `-vg2-lvol0   253:3    0  120M  0 lvm  
    `-vg3-lvol0 253:4    0  116M  0 lvm
Comment 5 Peter Rajnoha 2015-10-30 07:51:57 EDT
Yes, this doesn't work though (but that's a bug!):

0] raw/~ # vgcreate vg1 /dev/sda
  Physical volume "/dev/sda" successfully created
  Volume group "vg1" successfully created

[0] raw/~ # lvcreate -l100%FREE vg1
  Logical volume "lvol0" created.

[0] raw/~ # vgcreate vg2 /dev/vg1/lvol0
  Physical volume "/dev/vg1/lvol0" successfully created
  Volume group "vg2" successfully created

[0] raw/~ # vgchange -an vg1
  0 logical volume(s) in volume group "vg1" now active

[0] raw/~ # pvs
  WARNING: Device for PV MjuKj0-jw4k-Cbtt-PrWO-3XCc-yqK5-530dGv not found or rejected by a filter.
  WARNING: Device for PV MjuKj0-jw4k-Cbtt-PrWO-3XCc-yqK5-530dGv not found or rejected by a filter.
  PV             VG     Fmt  Attr PSize   PFree  
  /dev/sda       vg1    lvm2 a--  124.00m      0 
  /dev/vda2      fedora lvm2 a--   19.49g      0 
  unknown device vg2    lvm2 a-m  120.00m 120.00m


===

However, it works after reboot:

[0] raw/~ # vgs
  VG     #PV #LV #SN Attr   VSize   VFree  
  fedora   1   2   0 wz--n-  19.49g      0 
  vg1      1   1   0 wz--n- 124.00m      0 
  vg2      1   0   0 wz--n- 120.00m 120.00m

[0] raw/~ # vgchange -an vg1
  0 logical volume(s) in volume group "vg1" now active

[0] raw/~ # pvs
  PV         VG     Fmt  Attr PSize   PFree
  /dev/sda   vg1    lvm2 a--  124.00m    0 
  /dev/vda2  fedora lvm2 a--   19.49g    0 


I'll have a look...
Comment 6 Peter Rajnoha 2015-10-30 08:12:15 EDT
Ah yes, I know what's wrong!

The issue here is that there's no lvm2-pvscan@major:minor.service for the LV that acts as a PV:

[0] raw/~ # vgcreate vg1 /dev/sda
  Physical volume "/dev/sda" successfully created
  Volume group "vg1" successfully created

[0] raw/~ # lvcreate -l100%FREE vg1 
  Logical volume "lvol0" created.

[0] raw/~ # vgcreate vg2 /dev/vg1/lvol0
  Physical volume "/dev/vg1/lvol0" successfully created
  Volume group "vg2" successfully created

[0] raw/~ # lvs -o name,kernel_major,kernel_minor vg1/lvol0
  LV    KMaj KMin
  lvol0  253    2

[0] raw/~ # systemctl status lvm2-pvscan@253:2.service
* lvm2-pvscan@253:2.service - LVM2 PV scan on device 253:2
   Loaded: loaded (/usr/lib/systemd/system/lvm2-pvscan@.service; static; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:pvscan(8)

(...reboot...)

0] raw/~ # systemctl status lvm2-pvscan@253:2.service
* lvm2-pvscan@253:2.service - LVM2 PV scan on device 253:2
   Loaded: loaded (/usr/lib/systemd/system/lvm2-pvscan@.service; static; vendor preset: disabled)
   Active: active (exited) since Fri 2015-10-30 13:10:04 CET; 11s ago
     Docs: man:pvscan(8)
  Process: 662 ExecStart=/usr/sbin/lvm pvscan --cache --activate ay %i (code=exited, status=0/SUCCESS)
 Main PID: 662 (code=exited, status=0/SUCCESS)

Oct 30 13:10:04 raw.virt systemd[1]: Starting LVM2 PV scan on device 253:2...
Oct 30 13:10:04 raw.virt lvm[662]: 0 logical volume(s) in volume group "vg2" now active
Oct 30 13:10:04 raw.virt systemd[1]: Started LVM2 PV scan on device 253:2.



This means, there's no hook to call pvscan --cache major:minor for the removed LV (that is a PV at the same time) - this is normally executed as the lvm2-pvscan@major:minor.service's ExecStop action. But since we don't have the service...
Comment 7 Jan Kurik 2016-02-24 08:53:02 EST
This bug appears to have been reported against 'rawhide' during the Fedora 24 development cycle.
Changing version to '24'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora24#Rawhide_Rebase
Comment 8 Fedora End Of Life 2017-07-25 15:24:36 EDT
This message is a reminder that Fedora 24 is nearing its end of life.
Approximately 2 (two) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 24. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '24'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 24 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Note You need to log in before you can comment on or make changes to this bug.