Bug 1271291

Summary: pvmove can cause 'VG lock skipped: storage errors for sanlock leases'
Product: Red Hat Enterprise Linux 7 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: David Teigland <teigland>
lvm2 sub component: LVM lock daemon / lvmlockd QA Contact: cluster-qe <cluster-qe>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: high CC: agk, coughlan, heinzm, jbrassow, jkachuck, prajnoha, rbednar, teigland, zkabelac
Version: 7.2   
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: lvm2-2.02.156-1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 04:11:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1295577, 1313485, 1364088    

Description Corey Marthaler 2015-10-13 14:34:19 UTC
Description of problem:
I've been seeing this error a lot when attempting to clean up shared lvmlocd volumes lately.

# All LVs in VG snapper_thinp have been removed

harding-02: vgchange --lock-stop  snapper_thinp
  VG snapper_thinp lock failed: storage errors for sanlock leases

harding-03: vgremove  snapper_thinp

harding-02: vgchange --lock-stop  global

removing VG global on harding-03
harding-03: vgremove  global


# Another  shared VG, and again no LVs remain on it

removing VG snapper_thinp on mckinley-03
  VG snapper_thinp lock skipped: storage errors for sanlock leases
  Reading VG snapper_thinp without a lock.
mckinley-01: vgchange --lock-stop  snapper_thinp
  VG snapper_thinp lock failed: storage errors for sanlock leases
mckinley-02: vgchange --lock-stop  snapper_thinp
  VG snapper_thinp lock failed: storage errors for sanlock leases
mckinley-04: vgchange --lock-stop  snapper_thinp
  Cannot access VG snapper_thinp due to failed lock.
mckinley-03: vgremove  snapper_thinp
  VG snapper_thinp lock failed: storage errors for sanlock leases
unable to remove snapper_thinp on mckinley-03

# When i check dmsetup, often times the lvmlock device remains on other nodes even after the locks have been stopped.

[root@mckinley-01 ~]# dmsetup ls
mpathe  (253:2)
mpathe1 (253:9)
mpathd  (253:3)
mpathc  (253:12)
mpathb  (253:5)
mpathh1 (253:17)
mpathc1 (253:18)
mpatha  (253:4)
global-lvmlock  (253:19)
mpathf1 (253:16)
mpatha1 (253:14)
mpathd1 (253:8)
mpathh  (253:11)
snapper_thinp-lvmlock   (253:20)
mpathg  (253:7)
mpathg1 (253:15)
mpathb1 (253:13)
mpathf  (253:6)



Version-Release number of selected component (if applicable):
3.10.0-306.el7.x86_64

lvm2-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
lvm2-libs-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
lvm2-cluster-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-libs-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-event-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-event-libs-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-persistent-data-0.5.5-1.el7    BUILT: Thu Aug 13 09:58:10 CDT 2015
cmirror-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
sanlock-3.2.4-1.el7    BUILT: Fri Jun 19 12:48:49 CDT 2015
sanlock-lib-3.2.4-1.el7    BUILT: Fri Jun 19 12:48:49 CDT 2015
lvm2-lockd-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015


How reproducible:
Often

Comment 2 Corey Marthaler 2016-01-12 17:36:59 UTC
Quick note that this appears to be caused by pvmoving a device that also happens to contain the internal [lvmlock] device.

One of the test cases causing this:

SCENARIO - [thin_pool_pvmove]
Create snapshots of origin with fs data, and then pvmove the data from the thin pool volume
Making pool volume
lvcreate --activate ey --thinpool POOL  --zero n -L 4G --poolmetadatasize 4M snapper_thinp

Skipping meta check until supported with shared storage (bug 1265768)

Making origin volume
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n origin
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other1
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other2
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other3
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n other4
  WARNING: Sum of all thin volume sizes (5.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (4.00 GiB)!
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other5
  WARNING: Sum of all thin volume sizes (6.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (4.00 GiB)!
Placing an xfs filesystem on origin volume
Mounting origin volume

Writing files to /mnt/origin

Checking files on /mnt/origin

syncing before snap creation...
Making 1st snapshot of origin volume
lvcreate --activate ey -k n -s /dev/snapper_thinp/origin -n snap1
Moving data from thin pool volume...
pvmove /dev/mapper/mpathb1
Mounting 1st snap volume
Checking files on /mnt/snap1

Writing files to /mnt/origin

syncing before snap creation...
Making 2nd snapshot of origin volume
lvcreate --activate ey -k n -s /dev/snapper_thinp/origin -n snap2
Again, moving data from thin pool volume...
pvmove /dev/mapper/mpathc1
Mounting 2nd snap volume
Checking files on /mnt/snap2

umount /mnt/snap1 /mnt/snap2 /mnt/origin
Removing snap volume snapper_thinp/snap1
lvremove -f /dev/snapper_thinp/snap1
Removing snap volume snapper_thinp/snap2
lvremove -f /dev/snapper_thinp/snap2
Removing thin origin and other virtual thin volumes
Removing thinpool snapper_thinp/POOL


8 disk(s) to be used:
        harding-02=/dev/mapper/mpathg /dev/mapper/mpathf /dev/mapper/mpathd /dev/mapper/mpathe /dev/mapper/mpathh /dev/mapper/mpathc /dev/mapper/mpathb /dev/mapper/mpatha
        harding-03=/dev/mapper/mpathg /dev/mapper/mpathe /dev/mapper/mpathh /dev/mapper/mpathf /dev/mapper/mpathc /dev/mapper/mpathb /dev/mapper/mpathd /dev/mapper/mpatha
removing VG snapper_thinp on harding-03
  VG snapper_thinp lock skipped: storage errors for sanlock leases
  Reading VG snapper_thinp without a lock.
harding-02: vgchange --lock-stop  snapper_thinp
harding-03: vgremove  snapper_thinp
  VG snapper_thinp lock failed: storage errors for sanlock leases
unable to remove snapper_thinp on harding-03

Comment 3 David Teigland 2016-01-12 17:55:47 UTC
I've pushed out this commit that disallows pvmoving a PV under the lvmlock LV:
https://git.fedorahosted.org/cgit/lvm2.git/commit/?id=0c6946b4ce2fc664a66cd68a4fb2324bfb69b5ab

Comment 4 David Teigland 2016-01-19 15:59:21 UTC
*** Bug 1271299 has been marked as a duplicate of this bug. ***

Comment 7 Roman Bednář 2016-06-29 13:51:57 UTC
Verified. LVM now prevents 'pvmove' from devices under [lvmlock] lv.


# lvs -a -o +devices
  LV              VG            Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices       
  root            rhel_virt-150 -wi-ao----   6.99g                                                     /dev/vda2(214)
  swap            rhel_virt-150 -wi-ao---- 856.00m                                                     /dev/vda2(0)  
  POOL            vg            twi---t---   4.00g                                                     POOL_tdata(0) 
  [POOL_tdata]    vg            Twi-------   4.00g                                                     /dev/sda(65)  
  [POOL_tmeta]    vg            ewi-------   4.00m                                                     /dev/sdj(0)   
  [lvmlock]       vg            -wi-ao---- 256.00m                                                     /dev/sda(0)   
  [lvol0_pmspare] vg            ewi-------   4.00m                                                     /dev/sda(64)  


# pvmove /dev/sda /dev/sdb
  Cannot pvmove device /dev/sda used for sanlock leases.



3.10.0-445.el7.x86_64

lvm2-2.02.156-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
lvm2-libs-2.02.156-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
lvm2-cluster-2.02.156-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
device-mapper-1.02.126-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
device-mapper-libs-1.02.126-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
device-mapper-event-1.02.126-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
device-mapper-event-libs-1.02.126-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
device-mapper-persistent-data-0.6.2-0.1.rc8.el7    BUILT: Wed May  4 09:56:34 CEST 2016
cmirror-2.02.156-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
sanlock-3.4.0-1.el7    BUILT: Fri Jun 10 18:41:03 CEST 2016
sanlock-lib-3.4.0-1.el7    BUILT: Fri Jun 10 18:41:03 CEST 2016
lvm2-lockd-2.02.156-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016

Comment 9 errata-xmlrpc 2016-11-04 04:11:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1445.html