Bug 837599

Summary: [lvmetad] vgscan --cache does not flush information in lvmetad first
Product: Red Hat Enterprise Linux 6 Reporter: Marian Csontos <mcsontos>
Component: lvm2Assignee: Petr Rockai <prockai>
Status: CLOSED ERRATA QA Contact: Cluster QE <mspqa-list>
Severity: unspecified Docs Contact:
Priority: high    
Version: 6.3CC: agk, coughlan, dwysocha, heinzm, jbrassow, msnitzer, nperic, prajnoha, prockai, thornber, zkabelac
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: lvm2-2.02.98-1.el6 Doc Type: Bug Fix
Doc Text:
Issuing vgscan --cache (to refresh lvmetad) did not remove data about PVs or VGs that no longer exist, only updating metadata of existing entities. This has been fixed, and vgscan --cache will remove any metadata that is no longer relevant.
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-21 08:11:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marian Csontos 2012-07-04 10:51:19 UTC
Description of problem:
Running `vgscan --cache` does not remove metadata from lvmetad before scanning. To reread metadata one has to restart lvmetad or to call `pvscan --cache`.

Version-Release number of selected component (if applicable):
lvm2-2.02.96-10.el6

How reproducible:
100%

Steps to Reproduce:

# running lvmetad is assumed
PV=/dev/loop0
pvcreate $PV
vgcreate vgcorrupted $PV
lvcreate -n lvcorrupted -L 10M vgcorrupted
vgscan --cache
# corrupt MDA:
dd if=/dev/zero of=$PV count=16000
vgscan --cache

Actual results:
The VG is not listed on the second attempt, but PV, VG and LV are still listed in `pvs`, `vgs` and `lvs` output.
Writing to such corrupted MDA will correctly fail.
Only `pvscan --cache` will remove them;
dm device is left behind - run `dmsetup remove vgcorrupted-lvcorrupted` to clean up.

Expected results:
Error?

Additional info:

Comment 2 Peter Rajnoha 2012-07-04 11:24:31 UTC
There's also a problem with pvscan (probably the same or very related to the problem already reported here):

If, for some reason (I'm still investigating why), I end up with:

[1] alatyr/~ # pvs
  No device found for PV OAB20Z-HCz7-gxbJ-0vX1-cJV6-uVvu-lxQe2r.
  No device found for PV gct2EQ-gbTi-WPnl-4Eh1-YuP0-51KX-PL69FO.
  No device found for PV rSVXb0-9i7G-8Ra3-60N8-40bh-9P3I-7r6OuD.
  No device found for PV pf6EM0-nlZB-yXgl-7JSH-0nct-1Xby-kmdrha.
  No device found for PV i2BcEj-ILeH-7ZeW-jzbb-5tFw-uFn0-StlnE1.
  No device found for PV XaFv1W-PI0X-q3Da-xP7v-EHfx-TEYp-TBKJW1.
  No device found for PV 0cq5aT-ZHCi-kQse-0Z8z-omSE-cf2d-5mvdBk.
  No device found for PV ULblbL-AS9N-T5xj-hcuL-tfsX-9NNE-WBjMDG.
  No device found for PV ZuHtl3-9HIa-ZojI-rn2R-kleG-XHnB-Nha8QJ.
  No device found for PV ACGdB0-lB1R-qeUw-pBrk-NeHa-f9kq-bRLID0.
  No device found for PV iSvMu3-TtV6-aLwV-cYoE-HL9J-ik5q-jOlrVI.
  No device found for PV tyfM1X-AMtO-nKP0-OjJi-e8zQ-p0mn-Q0pp68.
  No device found for PV C4gXfA-shPV-WpBX-sUUp-GdKc-z2st-FoB2bW.
  No device found for PV ue5OmP-ctJ1-1NrV-CuLg-nDdM-FhZK-RXOOdR.
  No device found for PV Tm5fL1-xcTg-PA3r-R3xw-yOVU-BfUy-hw1TPG.
  No device found for PV fJanDp-3D3c-xhqS-mUVe-ND8e-b7OF-aBRMwI.
  No device found for PV C3KaCT-9xxj-Pddf-HNql-Rqff-9lQg-Zdk0B0.
  No device found for PV GRqDtx-NVvk-CveU-TbY0-6how-USq7-80M2DT.
  No device found for PV cs3aNN-LhF2-Qe0p-3oIJ-ZbRY-OG3e-woccJ3.
  No device found for PV 8hHsHu-EgoK-00x1-CzVe-vd8K-J0jy-d0AYQV.
  No device found for PV QoE3k2-XSjp-8kvp-ZJoi-qhVx-Hv62-Wspn6R.
  No device found for PV u6WXUh-F6wh-luNz-uAN1-AV7u-JKIP-KRY02k.
  No device found for PV yDytGb-UjOH-byIH-yOts-QFqC-uLps-9921dS.
  No device found for PV j0C2sX-4a6R-lFcA-66bA-hg3W-STc2-9nOzIl.
  No device found for PV XL5a6z-9FJe-jECf-cHdI-RPXb-A02l-YJxuFl.
  No device found for PV Gf4jOE-dcIF-PQSW-MP95-vTvB-TRXM-gqd5H9.
  No device found for PV lsfnKk-4TCe-g9MF-Cdo8-td0q-rY3D-nzGpTM.
  No device found for PV 5bZSuj-0Mvo-a86k-H2io-tkDH-TULc-LnI9O6.
  No device found for PV f52muY-i8Td-1T6y-bbtQ-216j-QXYO-NKfVfM.
  No device found for PV r9oU3P-vTgK-OelU-8aqn-Fhhb-5yrd-7HHWzr.
  No device found for PV cbU1Pb-3Nsl-uAMw-Tjqq-mizd-RqWT-ip6RIt.
  No device found for PV 3bWaWD-EJTg-HKhX-69W5-Bacf-BLmI-hCN1WK.
  No device found for PV Tq0YzT-OlyZ-iaHw-02dd-4Xgt-gVNK-4kBdH2.
  No device found for PV I1QBQr-Eu3D-L2e6-zFTJ-psbg-1O49-cTJHJM.
  No device found for PV jTGYfV-W26N-cV8H-H0lj-yA7k-vZ9r-Le84ck.
  No device found for PV 2qDPjb-dsC2-zk8F-yrtY-GFSk-ZfqR-OBs0kz.
  No device found for PV 87efLc-j2vd-vr6V-mimS-QrBZ-iuMg-P7A58e.
  No device found for PV vI0WP2-b1Um-ptGq-83CD-2bBR-JpW8-cm4icz.
  No device found for PV SQlkg0-ksqe-qDtx-zzFg-y2eW-ycFb-rOXCXS.
  No device found for PV HOYa0A-AFpF-Y4Jf-giFo-ESsF-XJHS-r0VmCq.
  No device found for PV hPJIJb-vGqW-iQ3H-vAl8-2ozs-HNvw-5fOBf6.
  No device found for PV 4o7CI8-SnZk-wPF0-53gN-PCpJ-COns-4MM9wf.
  No device found for PV 1d8744-izLP-L1Mw-uU1c-9rDj-Wu9J-3bQGtM.
  No device found for PV N3Gj7m-ZHeP-5W1R-B07p-lcGa-7luM-v1wmDS.
  No device found for PV QoBCmw-mlpu-etqi-L2ke-j7DF-CziG-NkePpY.
  No device found for PV R262xZ-kzAA-U03e-cWB8-qInm-L8H3-cTknVP.
  No device found for PV l33QvS-qp24-Meo0-sq8F-KQNF-XkP2-WXdkGL.
  No device found for PV UN5RTq-yXPf-8qhY-xSwF-QOPn-FWQ0-UiLPLH.
  No device found for PV s4AOwi-Tu24-wsiH-y7Yp-3H4U-kG4N-bcD0Sz.
  No device found for PV Ba2Xe2-1o0B-4mzu-77uN-xftD-iCOx-R7B6dE.
  No device found for PV PBQppR-6xc8-eAzB-I4n5-hXrM-vWev-xwz5G9.
  No device found for PV QUHwNG-1c7p-cr5N-rGUo-d1aw-NfXU-WkKPiI.
  No device found for PV nbgJGa-6sH7-STwv-rSg2-FAHg-nBb6-XVpBoR.
  No device found for PV EEO9ia-6W9P-mBLP-49fy-TVrf-mJV6-OsZwx6.
  No device found for PV oTDFd7-0xGj-vvQO-UiPu-Lg4Q-mMQh-ZHqliD.
  No device found for PV s6uTev-cku5-Jeme-2PGA-YWmL-an9G-dKY2bI.
  No device found for PV uI29YQ-RVlk-c2w0-3DLW-smiA-klOG-S49MPe.
  No device found for PV lJTPOl-rSe3-bPe7-OmLs-ICID-eWf7-kDDpOL.
  No device found for PV NcZ2pN-qAoA-o0la-Vmhi-2Oez-tR0H-D7gniw.
  No device found for PV UY6yZd-Yity-ZiMM-vSTO-2uT8-sa4h-fdBx9G.
  No device found for PV cvGZaL-XA1K-1TVs-e9A9-kiIf-6HC1-wkOh7J.
  No device found for PV 1OIJrt-YcLY-g0d1-s3ut-By1u-8HsI-IHS1aY.
  No device found for PV CnRROT-wJTc-Xfep-UMzw-mGf3-1F9j-heeW4s.
  PV                                                    VG        Fmt  Attr PSize   PFree 
  /dev/mapper/luks-58353f0b-7f26-4b86-a087-59dfb1b1d6db vg_alatyr lvm2 a--   99.97g     0 
  /dev/sda3                                             vg_data   lvm2 a--  365.26g 50.13g

[1] alatyr/~ # pvscan --cache
  No PV label found on /dev/sda1.
  No PV label found on /dev/vg_alatyr/lv_root.
  No PV label found on /dev/sda2.
  No PV label found on /dev/vg_alatyr/lv_swap.
  No PV label found on /dev/vg_alatyr/lv_home.

[1] alatyr/~ # pvs
  No device found for PV OAB20Z-HCz7-gxbJ-0vX1-cJV6-uVvu-lxQe2r.
  No device found for PV gct2EQ-gbTi-WPnl-4Eh1-YuP0-51KX-PL69FO.
  No device found for PV rSVXb0-9i7G-8Ra3-60N8-40bh-9P3I-7r6OuD.
  etc...

The "pvscan --cache" should have flushed all incorrect information and the "pvs" called afterwards should have clean info...

Comment 3 Peter Rajnoha 2012-07-04 11:34:41 UTC
An excerpt from the -vvvv log of the pvs command - the regex is correctly skipping the device, however, lvmetad code is still trying to access it:

#filters/filter-regex.c:173         /dev/vg_data/rhel6_stg_shared_01: Skipping (regex)
#cache/lvmetad.c:124   No device found for PV gct2EQ-gbTi-WPnl-4Eh1-YuP0-51KX-PL69FO.
#libdm-config.c:758       Setting id to rSVXb0-9i7G-8Ra3-60N8-40bh-9P3I-7r6OuD
#libdm-config.c:758       Setting format to lvm2
#libdm-config.c:789       Setting device to 64821
#libdm-config.c:789       Setting dev_size to 134217728
#libdm-config.c:789       Setting label_sector to 1

Comment 4 Peter Rajnoha 2012-07-04 12:32:35 UTC
So two issues here actually:

1.
"pvscan --cache" uses filter, but "pvscan --cache <device>" *does not* use filtering! But if any block device appears on a system (including device-mapper devices - LVs, in my case it was an LV used for a guest where inside this LV a PV was defined, but only to be visible for a guest, not for the host - so I set that in host's lvm.conf filter), the "pvscan --cache <device>" is called from within 69-dm-lvmetad.rules by default on *all block devices that are marked as PVs* (the scan whether this is a PV or not is done by blkid).

So "pvscan --cache <device>" should probably use filters as well!

There's probably a counterargument that if one uses the <device> directly on the command line, we don't need to filter as this is user's choice to directly scan the device... but we call "pvscan --cache <device>" in udev rules and we'd need to read the lvm.conf and do the filtering outside the pvscan --cache call otherwise and that would not be practical nor effective (this would require another call in udev rules, just to read the lvm.conf filters.

2.
The vgscan/pvscan --cache issue and the old information not being flushed from lvmetad if it is detected to be stale within the vgscan/pvscan --cache call.

Comment 5 Petr Rockai 2012-09-10 14:18:10 UTC
In fact, vgscan --cache needs to be dropped (or made an alias to pvscan --cache). The code is bogus. The filter issues are tracked in 814782 (should be ready to merge). I'll see to both.

Comment 6 Petr Rockai 2012-09-26 18:44:01 UTC
Fixed upstream.

Comment 8 Nenad Peric 2013-01-22 14:23:14 UTC
Testing using the reported instructions:

(08:20:21) [root@r6-node01:~]$ pvcreate /dev/sdc1
  Physical volume "/dev/sdc1" successfully created
(08:20:25) [root@r6-node01:~]$ vgcreate corrupted /dev/sdc1
  Volume group "corrupted" successfully created
(08:20:34) [root@r6-node01:~]$ lvcreate -n lvcorr corrupted -l 9
  Logical volume "lvcorr" created
(08:20:46) [root@r6-node01:~]$ vgscan --cache
  Reading all physical volumes.  This may take a while...
  Found volume group "corrupted" using metadata type lvm2
  Found volume group "VolGroup" using metadata type lvm2
(08:20:52) [root@r6-node01:~]$ lvs
  LV      VG        Attr      LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  lv_root VolGroup  -wi-ao---  7.54g                                             
  lv_swap VolGroup  -wi-ao---  1.97g                                             
  lvcorr  corrupted -wi-a---- 36.00m                                             
(08:20:55) [root@r6-node01:~]$ pvs
  PV         VG        Fmt  Attr PSize  PFree 
  /dev/sdb1            lvm2 a--  10.00g 10.00g
  /dev/sdc1  corrupted lvm2 a--   9.99g  9.96g
  /dev/sdd1            lvm2 a--  10.00g 10.00g
  /dev/sdf1            lvm2 a--  10.00g 10.00g
  /dev/vda2  VolGroup  lvm2 a--   9.51g     0 
(08:20:55) [root@r6-node01:~]$ dd if=/dev/urandom of=/dev/sdc1 bs=1024 count=10240
10240+0 records in
10240+0 records out
10485760 bytes (10 MB) copied, 1.27014 s, 8.3 MB/s
(08:21:10) [root@r6-node01:~]$ vgscan --cache
  Reading all physical volumes.  This may take a while...
  Found volume group "VolGroup" using metadata type lvm2
(08:21:13) [root@r6-node01:~]$ lvs
  LV      VG       Attr      LSize Pool Origin Data%  Move Log Cpy%Sync Convert
  lv_root VolGroup -wi-ao--- 7.54g                                             
  lv_swap VolGroup -wi-ao--- 1.97g                                             
(08:21:16) [root@r6-node01:~]$ vgs
  VG       #PV #LV #SN Attr   VSize VFree
  VolGroup   1   2   0 wz--n- 9.51g    0 
(08:21:17) [root@r6-node01:~]$ pvs
  PV         VG       Fmt  Attr PSize  PFree 
  /dev/sdb1           lvm2 a--  10.00g 10.00g
  /dev/sdd1           lvm2 a--  10.00g 10.00g
  /dev/sdf1           lvm2 a--  10.00g 10.00g
  /dev/vda2  VolGroup lvm2 a--   9.51g     0 
(08:21:18) [root@r6-node01:~]$ dmsetup ls
VolGroup-lv_swap	(253:1)
VolGroup-lv_root	(253:0)
corrupted-lvcorr	(253:2)
(08:21:21) [root@r6-node01:~]$ dmsetup remove corrupted-lvcorr
(08:21:28) [root@r6-node01:~]$ 


The cleanup of a leftover LV has to be done with dmsetup in the end as described in first comment as well.


Verified with:

lvm2-2.02.98-8.el6.x86_64
lvm2-libs-2.02.98-8.el6.x86_64
device-mapper-1.02.77-8.el6.x86_64
udev-147-2.46.el6.x86_64
kernel-2.6.32-355.el6.x86_64

Comment 9 errata-xmlrpc 2013-02-21 08:11:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0501.html