Bug 1271291 - pvmove can cause 'VG lock skipped: storage errors for sanlock leases'
pvmove can cause 'VG lock skipped: storage errors for sanlock leases'
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: lvm2 (Show other bugs)
7.2
x86_64 Linux
high Severity medium
: rc
: ---
Assigned To: David Teigland
cluster-qe@redhat.com
:
: 1271299 (view as bug list)
Depends On:
Blocks: 1313485 1295577 1364088
  Show dependency treegraph
 
Reported: 2015-10-13 10:34 EDT by Corey Marthaler
Modified: 2016-11-04 00:11 EDT (History)
9 users (show)

See Also:
Fixed In Version: lvm2-2.02.156-1.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-11-04 00:11:28 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Corey Marthaler 2015-10-13 10:34:19 EDT
Description of problem:
I've been seeing this error a lot when attempting to clean up shared lvmlocd volumes lately.

# All LVs in VG snapper_thinp have been removed

harding-02: vgchange --lock-stop  snapper_thinp
  VG snapper_thinp lock failed: storage errors for sanlock leases

harding-03: vgremove  snapper_thinp

harding-02: vgchange --lock-stop  global

removing VG global on harding-03
harding-03: vgremove  global


# Another  shared VG, and again no LVs remain on it

removing VG snapper_thinp on mckinley-03
  VG snapper_thinp lock skipped: storage errors for sanlock leases
  Reading VG snapper_thinp without a lock.
mckinley-01: vgchange --lock-stop  snapper_thinp
  VG snapper_thinp lock failed: storage errors for sanlock leases
mckinley-02: vgchange --lock-stop  snapper_thinp
  VG snapper_thinp lock failed: storage errors for sanlock leases
mckinley-04: vgchange --lock-stop  snapper_thinp
  Cannot access VG snapper_thinp due to failed lock.
mckinley-03: vgremove  snapper_thinp
  VG snapper_thinp lock failed: storage errors for sanlock leases
unable to remove snapper_thinp on mckinley-03

# When i check dmsetup, often times the lvmlock device remains on other nodes even after the locks have been stopped.

[root@mckinley-01 ~]# dmsetup ls
mpathe  (253:2)
mpathe1 (253:9)
mpathd  (253:3)
mpathc  (253:12)
mpathb  (253:5)
mpathh1 (253:17)
mpathc1 (253:18)
mpatha  (253:4)
global-lvmlock  (253:19)
mpathf1 (253:16)
mpatha1 (253:14)
mpathd1 (253:8)
mpathh  (253:11)
snapper_thinp-lvmlock   (253:20)
mpathg  (253:7)
mpathg1 (253:15)
mpathb1 (253:13)
mpathf  (253:6)



Version-Release number of selected component (if applicable):
3.10.0-306.el7.x86_64

lvm2-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
lvm2-libs-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
lvm2-cluster-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-libs-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-event-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-event-libs-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-persistent-data-0.5.5-1.el7    BUILT: Thu Aug 13 09:58:10 CDT 2015
cmirror-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
sanlock-3.2.4-1.el7    BUILT: Fri Jun 19 12:48:49 CDT 2015
sanlock-lib-3.2.4-1.el7    BUILT: Fri Jun 19 12:48:49 CDT 2015
lvm2-lockd-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015


How reproducible:
Often
Comment 2 Corey Marthaler 2016-01-12 12:36:59 EST
Quick note that this appears to be caused by pvmoving a device that also happens to contain the internal [lvmlock] device.

One of the test cases causing this:

SCENARIO - [thin_pool_pvmove]
Create snapshots of origin with fs data, and then pvmove the data from the thin pool volume
Making pool volume
lvcreate --activate ey --thinpool POOL  --zero n -L 4G --poolmetadatasize 4M snapper_thinp

Skipping meta check until supported with shared storage (bug 1265768)

Making origin volume
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n origin
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other1
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other2
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other3
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n other4
  WARNING: Sum of all thin volume sizes (5.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (4.00 GiB)!
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other5
  WARNING: Sum of all thin volume sizes (6.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (4.00 GiB)!
Placing an xfs filesystem on origin volume
Mounting origin volume

Writing files to /mnt/origin

Checking files on /mnt/origin

syncing before snap creation...
Making 1st snapshot of origin volume
lvcreate --activate ey -k n -s /dev/snapper_thinp/origin -n snap1
Moving data from thin pool volume...
pvmove /dev/mapper/mpathb1
Mounting 1st snap volume
Checking files on /mnt/snap1

Writing files to /mnt/origin

syncing before snap creation...
Making 2nd snapshot of origin volume
lvcreate --activate ey -k n -s /dev/snapper_thinp/origin -n snap2
Again, moving data from thin pool volume...
pvmove /dev/mapper/mpathc1
Mounting 2nd snap volume
Checking files on /mnt/snap2

umount /mnt/snap1 /mnt/snap2 /mnt/origin
Removing snap volume snapper_thinp/snap1
lvremove -f /dev/snapper_thinp/snap1
Removing snap volume snapper_thinp/snap2
lvremove -f /dev/snapper_thinp/snap2
Removing thin origin and other virtual thin volumes
Removing thinpool snapper_thinp/POOL


8 disk(s) to be used:
        harding-02=/dev/mapper/mpathg /dev/mapper/mpathf /dev/mapper/mpathd /dev/mapper/mpathe /dev/mapper/mpathh /dev/mapper/mpathc /dev/mapper/mpathb /dev/mapper/mpatha
        harding-03=/dev/mapper/mpathg /dev/mapper/mpathe /dev/mapper/mpathh /dev/mapper/mpathf /dev/mapper/mpathc /dev/mapper/mpathb /dev/mapper/mpathd /dev/mapper/mpatha
removing VG snapper_thinp on harding-03
  VG snapper_thinp lock skipped: storage errors for sanlock leases
  Reading VG snapper_thinp without a lock.
harding-02: vgchange --lock-stop  snapper_thinp
harding-03: vgremove  snapper_thinp
  VG snapper_thinp lock failed: storage errors for sanlock leases
unable to remove snapper_thinp on harding-03
Comment 3 David Teigland 2016-01-12 12:55:47 EST
I've pushed out this commit that disallows pvmoving a PV under the lvmlock LV:
https://git.fedorahosted.org/cgit/lvm2.git/commit/?id=0c6946b4ce2fc664a66cd68a4fb2324bfb69b5ab
Comment 4 David Teigland 2016-01-19 10:59:21 EST
*** Bug 1271299 has been marked as a duplicate of this bug. ***
Comment 7 Roman Bednář 2016-06-29 09:51:57 EDT
Verified. LVM now prevents 'pvmove' from devices under [lvmlock] lv.


# lvs -a -o +devices
  LV              VG            Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices       
  root            rhel_virt-150 -wi-ao----   6.99g                                                     /dev/vda2(214)
  swap            rhel_virt-150 -wi-ao---- 856.00m                                                     /dev/vda2(0)  
  POOL            vg            twi---t---   4.00g                                                     POOL_tdata(0) 
  [POOL_tdata]    vg            Twi-------   4.00g                                                     /dev/sda(65)  
  [POOL_tmeta]    vg            ewi-------   4.00m                                                     /dev/sdj(0)   
  [lvmlock]       vg            -wi-ao---- 256.00m                                                     /dev/sda(0)   
  [lvol0_pmspare] vg            ewi-------   4.00m                                                     /dev/sda(64)  


# pvmove /dev/sda /dev/sdb
  Cannot pvmove device /dev/sda used for sanlock leases.



3.10.0-445.el7.x86_64

lvm2-2.02.156-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
lvm2-libs-2.02.156-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
lvm2-cluster-2.02.156-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
device-mapper-1.02.126-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
device-mapper-libs-1.02.126-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
device-mapper-event-1.02.126-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
device-mapper-event-libs-1.02.126-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
device-mapper-persistent-data-0.6.2-0.1.rc8.el7    BUILT: Wed May  4 09:56:34 CEST 2016
cmirror-2.02.156-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
sanlock-3.4.0-1.el7    BUILT: Fri Jun 10 18:41:03 CEST 2016
sanlock-lib-3.4.0-1.el7    BUILT: Fri Jun 10 18:41:03 CEST 2016
lvm2-lockd-2.02.156-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
Comment 9 errata-xmlrpc 2016-11-04 00:11:28 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1445.html

Note You need to log in before you can comment on or make changes to this bug.