Bug 993942 - lvmetad keeps information about "unplugged" PV which was stacked on top of LV (LV removal)
Summary: lvmetad keeps information about "unplugged" PV which was stacked on top of LV...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: lvm2
Version: 20
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Peter Rajnoha
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-08-06 13:23 UTC by Peter Rajnoha
Modified: 2013-09-26 06:17 UTC (History)
11 users (show)

Fixed In Version: lvm2-2.02.102-1.fc20
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-09-26 06:17:15 UTC
Type: Bug


Attachments (Terms of Use)

Description Peter Rajnoha 2013-08-06 13:23:22 UTC
Reproducible with snapshots (but not with linears):

(virtualsize snapshots)
[0] raw/~ # pvs -o name,uuid
  PV         PV UUID                               
  /dev/vda2  WgtPbj-w1aa-2PDr-DcZq-8ijI-eVaZ-JwOqfy
[0] raw/~ # pvcreate /dev/sda
  Physical volume "/dev/sda" successfully created
[0] raw/~ # vgcreate vg /dev/sda
  Volume group "vg" successfully created
[0] raw/~ # lvcreate -s -l 100%FREE vg --virtualsize 1g
  Logical volume "lvol0" created
[0] raw/~ # pvs -o name,uuid
  PV         PV UUID                               
  /dev/sda   osIeTY-apW8-vQd3-5Gf1-XLej-QikC-Sg16S4
  /dev/vda2  WgtPbj-w1aa-2PDr-DcZq-8ijI-eVaZ-JwOqfy
[0] raw/~ # lvs -o name,uuid
  LV    LV UUID                               
  root  OhVTmn-Dbfi-Y0Wx-jiSN-gYVH-bxJK-B4K8c3
  swap  B9ZtZr-feUF-P8zW-tdIS-W61t-Scu5-o0bivZ
  lvol0 vBJjAq-t0Z4-MsnS-gRAv-flXI-BuVv-uOaeaD
[0] raw/~ # pvcreate /dev/vg/lvol0
  Physical volume "/dev/vg/lvol0" successfully created
[0] raw/~ # pvs -o name,uuid     
  PV            PV UUID                               
  /dev/sda      osIeTY-apW8-vQd3-5Gf1-XLej-QikC-Sg16S4
  /dev/vda2     WgtPbj-w1aa-2PDr-DcZq-8ijI-eVaZ-JwOqfy
  /dev/vg/lvol0 2s3zkB-eDLF-UUUg-uwAQ-o21c-Od13-63dOuK
[0] raw/~ # lvremove -ff vg/lvol0
  Logical volume "lvol0" successfully removed
[0] raw/~ # pvs -o name,uuid
  No device found for PV 2s3zkB-eDLF-UUUg-uwAQ-o21c-Od13-63dOuK.
  PV         PV UUID                               
  /dev/sda   osIeTY-apW8-vQd3-5Gf1-XLej-QikC-Sg16S4
  /dev/vda2  WgtPbj-w1aa-2PDr-DcZq-8ijI-eVaZ-JwOqfy
[0] raw/~ # pvscan --cache
[0] raw/~ # pvs -o name,uuid
  PV         PV UUID                               
  /dev/sda   osIeTY-apW8-vQd3-5Gf1-XLej-QikC-Sg16S4
  /dev/vda2  WgtPbj-w1aa-2PDr-DcZq-8ijI-eVaZ-JwOqfy



(normal snapshots)
[1] raw/~ # pvcreate /dev/sda
  Physical volume "/dev/sda" successfully created
[1] raw/~ # vgcreate vg /dev/sda
  Volume group "vg" successfully created
[1] raw/~ # lvcreate -L32m vg
  Logical volume "lvol0" created
[1] raw/~ # lvcreate -s vg/lvol0 -L 16m
  Logical volume "lvol1" created
[1] raw/~ # pvcreate vg/lvol1
  Device vg/lvol1 not found (or ignored by filtering).
[1] raw/~ # pvcreate /dev/vg/lvol1
  Physical volume "/dev/vg/lvol1" successfully created
[1] raw/~ # pvs -o name,uuid
  PV            PV UUID                               
  /dev/sda      OanVyN-YTFQ-WoHF-8M8u-jtOs-LJxM-b9zolZ
  /dev/vda2     WgtPbj-w1aa-2PDr-DcZq-8ijI-eVaZ-JwOqfy
  /dev/vg/lvol1 brLLPR-vF7n-222c-zGLa-FfCt-YmIY-c5JDfm
[1] raw/~ # lvremove -ff vg/lvol1 
  Logical volume "lvol1" successfully removed
[1] raw/~ # pvs -o name,uuid
  No device found for PV brLLPR-vF7n-222c-zGLa-FfCt-YmIY-c5JDfm.
  PV         PV UUID                               
  /dev/sda   OanVyN-YTFQ-WoHF-8M8u-jtOs-LJxM-b9zolZ
  /dev/vda2  WgtPbj-w1aa-2PDr-DcZq-8ijI-eVaZ-JwOqfy
[1] raw/~ # pvscan --cache
[1] raw/~ # pvs -o name,uuid
  PV         PV UUID                               
  /dev/sda   osIeTY-apW8-vQd3-5Gf1-XLej-QikC-Sg16S4
  /dev/vda2  WgtPbj-w1aa-2PDr-DcZq-8ijI-eVaZ-JwOqfy

Comment 1 Petr Rockai 2013-08-22 17:41:36 UTC
Could this be a problem with udev rules not firing on the snapshot, while they do fire on the linear? Also, does pvscan --cache <device> (I see it does without <device>) fix the problem?

Comment 2 Peter Rajnoha 2013-08-23 12:39:00 UTC
Looked into this more and I found out that the underlying LV on top of which the PV is stacked is not identified as "LVM2_member" during the REMOVE event. This causes that the pvscan --cache --major ... --minor ... is skipped. Looking at the event monitor, there's a CHANGE event just before REMOVE event for snapshots and it seems that this CHANGE event originates from clearing the PV label (as part of some cleanup) for which, as a consequence, the blkid clears the LVM2_member identification in udev db. I need to investigate more to be more precise about what's happening in detail...

Comment 3 Peter Rajnoha 2013-08-26 13:46:00 UTC
The patch:

https://git.fedorahosted.org/cgit/lvm2.git/commit/?id=4d3b5724e0b51782000a45027de00e0fed1c9833


    udev: inform lvmetad about lost PV label
    
    In stacked environment where we have a PV layered on top of a
    snapshot LV and then removing the LV, lvmetad still keeps information
    about the PV:
    
    [0] raw/~ $ pvcreate /dev/sda
      Physical volume "/dev/sda" successfully created
    [0] raw/~ $ vgcreate vg /dev/sda
      Volume group "vg" successfully created
    [0] raw/~ $ lvcreate -L32m vg
      Logical volume "lvol0" created
    [0] raw/~ $ lvcreate -L32m -s vg/lvol0
      Logical volume "lvol1" created
    [0] raw/~ $ pvcreate /dev/vg/lvol1
      Physical volume "/dev/vg/lvol1" successfully created
    [0] raw/~ $ lvremove -ff vg/lvol1
      Logical volume "lvol1" successfully removed
    [0] raw/~ $ pvs
      No device found for PV BdNlu2-7bHV-XcIp-mFFC-PPuR-ef6K-yffdzO.
      PV         VG         Fmt  Attr PSize   PFree
      /dev/sda   vg         lvm2 a--  124.00m 92.00m
    [0] raw/~ $ pvscan --cache --major 253 --minor 3
      Device 253:3 not found. Cleared from lvmetad cache.
    
    This is because of the reactivation that is done just before
    snapshot removal as part of the process (vg/lvol1 from the example above).
    This causes a CHANGE event to be generated, but any scan done
    on the LV does not see the original data anymore (in this case
    the stacked PV label on top) and consequently the ID_FS_TYPE="LVM2_member"
    (provided by blkid scan) is not stored in udev db anymore for the LV.
    Consequently, the pvscan --cache is not run anymore as the dev is not
    identified as LVM PV by the "LVM2_member" id - lvmetad loses this info
    and still keeps records about the PV.
    
    We can run into a very similar problem with erasing the PV label directly:
    
    [0] raw/~ $ lvcreate -L32m vg
      Logical volume "lvol0" created
    [0] raw/~ $ pvcreate /dev/vg/lvol0
      Physical volume "/dev/vg/lvol0" successfully created
    [0] raw/~ $ dd if=/dev/zero of=/dev/vg/lvol0 bs=1M
    dd: error writing '/dev/vg/lvol0': No space left on device
    33+0 records in
    32+0 records out
    33554432 bytes (34 MB) copied, 0.380921 s, 88.1 MB/s
    [0] raw/~ $ pvs
      PV            VG         Fmt  Attr PSize   PFree
      /dev/sda      vg         lvm2 a--  124.00m 92.00m
      /dev/vg/lvol0            lvm2 a--   32.00m 32.00m
    [0] raw/~ $ pvscan --cache --major 253 --minor 2
      No PV label found on /dev/vg/lvol0.
    
    This patch adds detection of this change from ID_FS_LABEL="LVM2_member"
    to ID_FS_LABEL="<whatever_else>" and hence informing the lvmetad
    about PV being gone.

Comment 4 Fedora End Of Life 2013-09-16 16:55:30 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 20 development cycle.
Changing version to '20'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora20

Comment 5 Fedora Update System 2013-09-24 08:47:10 UTC
lvm2-2.02.102-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/lvm2-2.02.102-1.fc20

Comment 6 Fedora Update System 2013-09-24 22:52:01 UTC
Package lvm2-2.02.102-1.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing lvm2-2.02.102-1.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-17514/lvm2-2.02.102-1.fc20
then log in and leave karma (feedback).

Comment 7 Fedora Update System 2013-09-26 06:17:15 UTC
lvm2-2.02.102-1.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.