RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1271291 - pvmove can cause 'VG lock skipped: storage errors for sanlock leases'
Summary: pvmove can cause 'VG lock skipped: storage errors for sanlock leases'
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: lvm2
Version: 7.2
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: rc
: ---
Assignee: David Teigland
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
: 1271299 (view as bug list)
Depends On:
Blocks: 1295577 1313485 1364088
TreeView+ depends on / blocked
 
Reported: 2015-10-13 14:34 UTC by Corey Marthaler
Modified: 2021-09-03 12:55 UTC (History)
9 users (show)

Fixed In Version: lvm2-2.02.156-1.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-04 04:11:28 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1445 0 normal SHIPPED_LIVE lvm2 bug fix and enhancement update 2016-11-03 13:46:41 UTC

Description Corey Marthaler 2015-10-13 14:34:19 UTC 
Description of problem:
I've been seeing this error a lot when attempting to clean up shared lvmlocd volumes lately.

# All LVs in VG snapper_thinp have been removed

harding-02: vgchange --lock-stop  snapper_thinp
  VG snapper_thinp lock failed: storage errors for sanlock leases

harding-03: vgremove  snapper_thinp

harding-02: vgchange --lock-stop  global

removing VG global on harding-03
harding-03: vgremove  global


# Another  shared VG, and again no LVs remain on it

removing VG snapper_thinp on mckinley-03
  VG snapper_thinp lock skipped: storage errors for sanlock leases
  Reading VG snapper_thinp without a lock.
mckinley-01: vgchange --lock-stop  snapper_thinp
  VG snapper_thinp lock failed: storage errors for sanlock leases
mckinley-02: vgchange --lock-stop  snapper_thinp
  VG snapper_thinp lock failed: storage errors for sanlock leases
mckinley-04: vgchange --lock-stop  snapper_thinp
  Cannot access VG snapper_thinp due to failed lock.
mckinley-03: vgremove  snapper_thinp
  VG snapper_thinp lock failed: storage errors for sanlock leases
unable to remove snapper_thinp on mckinley-03

# When i check dmsetup, often times the lvmlock device remains on other nodes even after the locks have been stopped.

[root@mckinley-01 ~]# dmsetup ls
mpathe  (253:2)
mpathe1 (253:9)
mpathd  (253:3)
mpathc  (253:12)
mpathb  (253:5)
mpathh1 (253:17)
mpathc1 (253:18)
mpatha  (253:4)
global-lvmlock  (253:19)
mpathf1 (253:16)
mpatha1 (253:14)
mpathd1 (253:8)
mpathh  (253:11)
snapper_thinp-lvmlock   (253:20)
mpathg  (253:7)
mpathg1 (253:15)
mpathb1 (253:13)
mpathf  (253:6)



Version-Release number of selected component (if applicable):
3.10.0-306.el7.x86_64

lvm2-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
lvm2-libs-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
lvm2-cluster-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-libs-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-event-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-event-libs-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-persistent-data-0.5.5-1.el7    BUILT: Thu Aug 13 09:58:10 CDT 2015
cmirror-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
sanlock-3.2.4-1.el7    BUILT: Fri Jun 19 12:48:49 CDT 2015
sanlock-lib-3.2.4-1.el7    BUILT: Fri Jun 19 12:48:49 CDT 2015
lvm2-lockd-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015


How reproducible:
Often
Comment 2 Corey Marthaler 2016-01-12 17:36:59 UTC 
Quick note that this appears to be caused by pvmoving a device that also happens to contain the internal [lvmlock] device.

One of the test cases causing this:

SCENARIO - [thin_pool_pvmove]
Create snapshots of origin with fs data, and then pvmove the data from the thin pool volume
Making pool volume
lvcreate --activate ey --thinpool POOL  --zero n -L 4G --poolmetadatasize 4M snapper_thinp

Skipping meta check until supported with shared storage (bug 1265768)

Making origin volume
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n origin
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other1
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other2
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other3
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n other4
  WARNING: Sum of all thin volume sizes (5.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (4.00 GiB)!
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other5
  WARNING: Sum of all thin volume sizes (6.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (4.00 GiB)!
Placing an xfs filesystem on origin volume
Mounting origin volume

Writing files to /mnt/origin

Checking files on /mnt/origin

syncing before snap creation...
Making 1st snapshot of origin volume
lvcreate --activate ey -k n -s /dev/snapper_thinp/origin -n snap1
Moving data from thin pool volume...
pvmove /dev/mapper/mpathb1
Mounting 1st snap volume
Checking files on /mnt/snap1

Writing files to /mnt/origin

syncing before snap creation...
Making 2nd snapshot of origin volume
lvcreate --activate ey -k n -s /dev/snapper_thinp/origin -n snap2
Again, moving data from thin pool volume...
pvmove /dev/mapper/mpathc1
Mounting 2nd snap volume
Checking files on /mnt/snap2

umount /mnt/snap1 /mnt/snap2 /mnt/origin
Removing snap volume snapper_thinp/snap1
lvremove -f /dev/snapper_thinp/snap1
Removing snap volume snapper_thinp/snap2
lvremove -f /dev/snapper_thinp/snap2
Removing thin origin and other virtual thin volumes
Removing thinpool snapper_thinp/POOL


8 disk(s) to be used:
        harding-02=/dev/mapper/mpathg /dev/mapper/mpathf /dev/mapper/mpathd /dev/mapper/mpathe /dev/mapper/mpathh /dev/mapper/mpathc /dev/mapper/mpathb /dev/mapper/mpatha
        harding-03=/dev/mapper/mpathg /dev/mapper/mpathe /dev/mapper/mpathh /dev/mapper/mpathf /dev/mapper/mpathc /dev/mapper/mpathb /dev/mapper/mpathd /dev/mapper/mpatha
removing VG snapper_thinp on harding-03
  VG snapper_thinp lock skipped: storage errors for sanlock leases
  Reading VG snapper_thinp without a lock.
harding-02: vgchange --lock-stop  snapper_thinp
harding-03: vgremove  snapper_thinp
  VG snapper_thinp lock failed: storage errors for sanlock leases
unable to remove snapper_thinp on harding-03
Comment 3 David Teigland 2016-01-12 17:55:47 UTC 
I've pushed out this commit that disallows pvmoving a PV under the lvmlock LV:
https://git.fedorahosted.org/cgit/lvm2.git/commit/?id=0c6946b4ce2fc664a66cd68a4fb2324bfb69b5ab
Comment 4 David Teigland 2016-01-19 15:59:21 UTC 
*** Bug 1271299 has been marked as a duplicate of this bug. ***
Comment 7 Roman Bednář 2016-06-29 13:51:57 UTC 
Verified. LVM now prevents 'pvmove' from devices under [lvmlock] lv.


# lvs -a -o +devices
  LV              VG            Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices       
  root            rhel_virt-150 -wi-ao----   6.99g                                                     /dev/vda2(214)
  swap            rhel_virt-150 -wi-ao---- 856.00m                                                     /dev/vda2(0)  
  POOL            vg            twi---t---   4.00g                                                     POOL_tdata(0) 
  [POOL_tdata]    vg            Twi-------   4.00g                                                     /dev/sda(65)  
  [POOL_tmeta]    vg            ewi-------   4.00m                                                     /dev/sdj(0)   
  [lvmlock]       vg            -wi-ao---- 256.00m                                                     /dev/sda(0)   
  [lvol0_pmspare] vg            ewi-------   4.00m                                                     /dev/sda(64)  


# pvmove /dev/sda /dev/sdb
  Cannot pvmove device /dev/sda used for sanlock leases.



3.10.0-445.el7.x86_64

lvm2-2.02.156-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
lvm2-libs-2.02.156-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
lvm2-cluster-2.02.156-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
device-mapper-1.02.126-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
device-mapper-libs-1.02.126-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
device-mapper-event-1.02.126-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
device-mapper-event-libs-1.02.126-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
device-mapper-persistent-data-0.6.2-0.1.rc8.el7    BUILT: Wed May  4 09:56:34 CEST 2016
cmirror-2.02.156-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
sanlock-3.4.0-1.el7    BUILT: Fri Jun 10 18:41:03 CEST 2016
sanlock-lib-3.4.0-1.el7    BUILT: Fri Jun 10 18:41:03 CEST 2016
lvm2-lockd-2.02.156-1.el7    BUILT: Mon Jun 13 10:05:51 CEST 2016
Comment 9 errata-xmlrpc 2016-11-04 04:11:28 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1445.html
Note You need to log in before you can comment on or make changes to this bug.