Bug 1271299

Summary: Global lock failed: error -218
Product: Red Hat Enterprise Linux 7 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: David Teigland <teigland>
lvm2 sub component: LVM lock daemon / lvmlockd QA Contact: cluster-qe <cluster-qe>
Status: CLOSED DUPLICATE Docs Contact:
Severity: medium    
Priority: unspecified CC: agk, heinzm, jbrassow, prajnoha, teigland, zkabelac
Version: 7.2   
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-19 15:59:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Corey Marthaler 2015-10-13 14:57:11 UTC 
Description of problem:
This is most likely related to (if not the exact same bug as) bug 1271291.

When exclusive locked LVs existed on VG, and those VGs are then removed, a VG can not be created on a different node, but can be on the same node that once held the exclusive lock.

 [root@harding-02 ~]# pvscan
  Skipping global lock: storage errors for sanlock leases
  PV /dev/sda2   VG rhel_harding-02   lvm2 [92.67 GiB / 60.00 MiB free]
  PV /dev/sdb1   VG rhel_harding-02   lvm2 [93.16 GiB / 0    free]
  PV /dev/sdc1   VG rhel_harding-02   lvm2 [93.16 GiB / 0    free]
  Total: 3 [278.98 GiB] / in use: 3 [278.98 GiB] / in no VG: 0 [0   ]

[root@harding-03 ~]# pvscan
  Skipping global lock: lockspace not found or started
  PV /dev/sda2   VG rhel_harding-03   lvm2 [92.67 GiB / 60.00 MiB free]
  PV /dev/sdb1   VG rhel_harding-03   lvm2 [93.16 GiB / 0    free]
  PV /dev/sdc1   VG rhel_harding-03   lvm2 [93.16 GiB / 0    free]
  Total: 3 [278.98 GiB] / in use: 3 [278.98 GiB] / in no VG: 0 [0   ]

[root@harding-02 ~]# dmsetup ls
mpathe  (253:7)
mpathe1 (253:17)
mpathd  (253:6)
mpathc  (253:5)
mpathb  (253:4)
mpathh1 (253:18)
mpathc1 (253:13)
mpatha  (253:3)
rhel_harding--02-home   (253:2)
mpathf1 (253:14)
mpatha1 (253:11)
rhel_harding--02-swap   (253:1)
rhel_harding--02-root   (253:0)
mpathd1 (253:16)
mpathh  (253:10)
mpathg  (253:8)
mpathg1 (253:12)
mpathb1 (253:15)
mpathf  (253:9)


[root@harding-03 ~]# dmsetup ls
mpathe  (253:9)
mpathe1 (253:17)
mpathd  (253:4)
mpathc  (253:5)
mpathb  (253:7)
mpathh1 (253:18)
mpathc1 (253:12)
mpatha  (253:6)
mpathf1 (253:16)
rhel_harding--03-home   (253:2)
mpatha1 (253:15)
mpathd1 (253:11)
rhel_harding--03-swap   (253:1)
rhel_harding--03-root   (253:0)
mpathh  (253:10)
mpathg  (253:3)
mpathg1 (253:13)
mpathb1 (253:14)
mpathf  (253:8)


[root@harding-02 ~]# systemctl status sanlock
â sanlock.service - Shared Storage Lease Manager
   Loaded: loaded (/usr/lib/systemd/system/sanlock.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2015-10-12 23:07:22 CDT; 15h ago
  Process: 3338 ExecStart=/lib/systemd/systemd-sanlock start (code=exited, status=0/SUCCESS)
 Main PID: 3356 (sanlock)
   CGroup: /system.slice/sanlock.service
           ââ3356 sanlock daemon -U sanlock -G sanlock
           ââ3357 sanlock daemon -U sanlock -G sanlock

Oct 13 14:12:12 harding-02.lab.msp.redhat.com sanlock[3356]: 2015-10-13 14:12:12-0500 55287 [3365]: r39 cmd_acquire 3,9,3369 invalid lockspace found -1 failed 0 name lvm_snapper_thinp
Oct 13 14:12:48 harding-02.lab.msp.redhat.com sanlock[3356]: 2015-10-13 14:12:48-0500 55323 [3364]: r40 cmd_acquire 3,9,3369 invalid lockspace found -1 failed 0 name lvm_snapper_thinp
Oct 13 14:12:50 harding-02.lab.msp.redhat.com sanlock[3356]: 2015-10-13 14:12:50-0500 55325 [3365]: r41 cmd_acquire 3,9,3369 invalid lockspace found -1 failed 0 name lvm_snapper_thinp
Oct 13 14:14:48 harding-02.lab.msp.redhat.com sanlock[3356]: 2015-10-13 14:14:48-0500 55443 [3364]: r42 cmd_acquire 3,9,3369 invalid lockspace found -1 failed 0 name lvm_snapper_thinp
Oct 13 14:15:56 harding-02.lab.msp.redhat.com sanlock[3356]: 2015-10-13 14:15:56-0500 55511 [3365]: r43 cmd_acquire 3,9,3369 invalid lockspace found -1 failed 0 name lvm_snapper_thinp
Oct 13 14:16:01 harding-02.lab.msp.redhat.com sanlock[3356]: 2015-10-13 14:16:01-0500 55516 [3364]: r44 cmd_acquire 3,9,3369 invalid lockspace found -1 failed 0 name lvm_snapper_thinp
Oct 13 14:35:37 harding-02.lab.msp.redhat.com sanlock[3356]: 2015-10-13 14:35:37-0500 56692 [3365]: r45 cmd_acquire 3,9,3369 invalid lockspace found -1 failed 0 name lvm_snapper_thinp
Oct 13 14:35:58 harding-02.lab.msp.redhat.com sanlock[3356]: 2015-10-13 14:35:58-0500 56712 [3364]: r46 cmd_acquire 3,9,3369 invalid lockspace found -1 failed 0 name lvm_snapper_thinp
Oct 13 14:39:51 harding-02.lab.msp.redhat.com sanlock[3356]: 2015-10-13 14:39:51-0500 56946 [3365]: r47 cmd_acquire 3,9,3369 invalid lockspace found -1 failed 0 name lvm_snapper_thinp
Oct 13 14:41:48 harding-02.lab.msp.redhat.com sanlock[3356]: 2015-10-13 14:41:48-0500 57063 [3364]: r48 cmd_acquire 3,9,3369 invalid lockspace found -1 failed 0 name lvm_snapper_thinp

[root@harding-03 ~]# systemctl status sanlock
â sanlock.service - Shared Storage Lease Manager
   Loaded: loaded (/usr/lib/systemd/system/sanlock.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2015-10-12 23:07:06 CDT; 15h ago
  Process: 3255 ExecStart=/lib/systemd/systemd-sanlock start (code=exited, status=0/SUCCESS)
 Main PID: 3273 (sanlock)
   CGroup: /system.slice/sanlock.service
           ââ3273 sanlock daemon -U sanlock -G sanlock
           ââ3274 sanlock daemon -U sanlock -G sanlock

Oct 12 23:07:05 harding-03.lab.msp.redhat.com systemd[1]: Starting Shared Storage Lease Manager...
Oct 12 23:07:06 harding-03.lab.msp.redhat.com systemd-sanlock[3255]: Starting sanlock: [  OK  ]
Oct 12 23:07:06 harding-03.lab.msp.redhat.com systemd[1]: Started Shared Storage Lease Manager.


[root@harding-02 ~]# systemctl status lvm2-lvmlockd
â lvm2-lvmlockd.service - LVM2 lock daemon
   Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmlockd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2015-10-12 23:07:23 CDT; 15h ago
     Docs: man:lvmlockd(8)
 Main PID: 3369 (lvmlockd)
   CGroup: /system.slice/lvm2-lvmlockd.service
           ââ3369 /usr/sbin/lvmlockd -f

Oct 13 14:16:01 harding-02.lab.msp.redhat.com lvmlockd[3369]: 1444763761 S lvm_snapper_thinp R GLLK lock_san acquire error -28
Oct 13 14:16:01 harding-02.lab.msp.redhat.com lvmlockd[3369]: 1444763761 S lvm_snapper_thinp R GLLK res_lock lm error -218
Oct 13 14:35:37 harding-02.lab.msp.redhat.com lvmlockd[3369]: 1444764937 S lvm_snapper_thinp R GLLK lock_san acquire error -28
Oct 13 14:35:37 harding-02.lab.msp.redhat.com lvmlockd[3369]: 1444764937 S lvm_snapper_thinp R GLLK res_lock lm error -218
Oct 13 14:35:58 harding-02.lab.msp.redhat.com lvmlockd[3369]: 1444764958 S lvm_snapper_thinp R GLLK lock_san acquire error -28
Oct 13 14:35:58 harding-02.lab.msp.redhat.com lvmlockd[3369]: 1444764958 S lvm_snapper_thinp R GLLK res_lock lm error -218
Oct 13 14:39:51 harding-02.lab.msp.redhat.com lvmlockd[3369]: 1444765191 S lvm_snapper_thinp R GLLK lock_san acquire error -28
Oct 13 14:39:51 harding-02.lab.msp.redhat.com lvmlockd[3369]: 1444765191 S lvm_snapper_thinp R GLLK res_lock lm error -218
Oct 13 14:41:48 harding-02.lab.msp.redhat.com lvmlockd[3369]: 1444765308 S lvm_snapper_thinp R GLLK lock_san acquire error -28
Oct 13 14:41:48 harding-02.lab.msp.redhat.com lvmlockd[3369]: 1444765308 S lvm_snapper_thinp R GLLK res_lock lm error -218


[root@harding-03 ~]# systemctl status lvm2-lvmlockd
â lvm2-lvmlockd.service - LVM2 lock daemon
   Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmlockd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2015-10-12 23:07:06 CDT; 15h ago
     Docs: man:lvmlockd(8)
 Main PID: 3286 (lvmlockd)
   CGroup: /system.slice/lvm2-lvmlockd.service
           ââ3286 /usr/sbin/lvmlockd -f

Oct 12 23:07:06 harding-03.lab.msp.redhat.com systemd[1]: Started LVM2 lock daemon.
Oct 12 23:07:06 harding-03.lab.msp.redhat.com systemd[1]: Starting LVM2 lock daemon...
Oct 12 23:07:06 harding-03.lab.msp.redhat.com lvmlockd[3286]: [D] creating /run/lvm/lvmlockd.socket
Oct 12 23:07:06 harding-03.lab.msp.redhat.com lvmlockd[3286]: 1444709226 lvmlockd started
Oct 12 23:56:20 harding-03.lab.msp.redhat.com lvmlockd[3286]: 1444712180 S lvm_snapper_thinp R VGLK res_update version on non-ex lock
Oct 12 23:56:22 harding-03.lab.msp.redhat.com lvmlockd[3286]: 1444712182 S lvm_snapper_thinp R VGLK res_update version on non-ex lock

[root@harding-02 ~]# vgcreate --shared global /dev/mapper/mpatha1
  WARNING: shared lock type "sanlock" and lvmlockd are Technology Preview.
  For more information on Technology Preview features, visit:
  https://access.redhat.com/support/offerings/techpreview/
  Global lock failed: error -218

[root@harding-03 ~]# vgcreate --shared global /dev/mapper/mpatha1
  WARNING: shared lock type "sanlock" and lvmlockd are Technology Preview.
  For more information on Technology Preview features, visit:
  https://access.redhat.com/support/offerings/techpreview/
  Enabling sanlock global lock
  Physical volume "/dev/mapper/mpatha1" successfully created
  Physical volume "/dev/mapper/mpatha1" successfully created
  Logical volume "lvmlock" created.
  Physical volume "/dev/mapper/mpatha1" successfully created
  Volume group "global" successfully created
  VG global starting sanlock lockspace
  Starting locking.  Waiting until locks are ready...


Version-Release number of selected component (if applicable):
3.10.0-306.el7.x86_64

lvm2-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
lvm2-libs-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
lvm2-cluster-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-libs-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-event-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-event-libs-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-persistent-data-0.5.5-1.el7    BUILT: Thu Aug 13 09:58:10 CDT 2015
cmirror-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
sanlock-3.2.4-1.el7    BUILT: Fri Jun 19 12:48:49 CDT 2015
sanlock-lib-3.2.4-1.el7    BUILT: Fri Jun 19 12:48:49 CDT 2015
lvm2-lockd-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
Comment 2 David Teigland 2016-01-19 15:59:21 UTC 

*** This bug has been marked as a duplicate of bug 1271291 ***