1610455 – sanlock VG creation failing w/o PV label on device to be used in vgcreate

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1610455 - sanlock VG creation failing w/o PV label on device to be used in vgcreate

Summary: sanlock VG creation failing w/o PV label on device to be used in vgcreate

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	lvm2
Sub Component:
Version:	7.6
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	David Teigland
QA Contact:	cluster-qe@redhat.com
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-07-31 16:31 UTC by Corey Marthaler
Modified:	2021-09-03 12:50 UTC (History)
CC List:	8 users (show)
Fixed In Version:	lvm2-2.02.180-4.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-10-30 11:03:51 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2018:3193	0	None	None	None	2018-10-30 11:04:56 UTC

Description Corey Marthaler 2018-07-31 16:31:12 UTC

Description of problem:
At first I thought this was a timing issue associated w/ the global lock only, but after attempting additional vgcreates, with the global lock in existence, those are failing as well. Also, even when creating the PV separately with " pvcreate --config global/use_lvmlockd=0 /dev/mapper/mpatha1" and having the vgcreate cmd pass, the vgremove still fails, so something bigger is wrong here.


[root@harding-02 ~]# systemctl status sanlock
â sanlock.service - Shared Storage Lease Manager
   Loaded: loaded (/usr/lib/systemd/system/sanlock.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-07-31 10:43:28 CDT; 30min ago
  Process: 52670 ExecStart=/usr/sbin/sanlock daemon (code=exited, status=0/SUCCESS)
 Main PID: 52671 (sanlock)
   CGroup: /system.slice/sanlock.service
           ââ52671 /usr/sbin/sanlock daemon
           ââ52672 /usr/sbin/sanlock daemon

Jul 31 10:43:28 harding-02.lab.msp.redhat.com systemd[1]: Starting Shared Storage Lease Manager...
Jul 31 10:43:28 harding-02.lab.msp.redhat.com systemd[1]: Started Shared Storage Lease Manager.

[root@harding-02 ~]# systemctl status lvm2-lvmlockd
â lvm2-lvmlockd.service - LVM2 lock daemon
   Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmlockd.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-07-31 10:43:29 CDT; 30min ago
     Docs: man:lvmlockd(8)
 Main PID: 52688 (lvmlockd)
   CGroup: /system.slice/lvm2-lvmlockd.service
           ââ52688 /usr/sbin/lvmlockd -f

Jul 31 10:43:29 harding-02.lab.msp.redhat.com systemd[1]: Started LVM2 lock daemon.
Jul 31 10:43:29 harding-02.lab.msp.redhat.com lvmlockd[52688]: [D] creating /run/lvm/lvmlockd.socket
Jul 31 10:43:29 harding-02.lab.msp.redhat.com lvmlockd[52688]: 1533051809 lvmlockd started

# without a PV created on /dev/mapper/mpatha1 yet
[root@harding-02 ~]# vgcreate --shared foo /dev/mapper/mpatha1 
  Enabling sanlock global lock
  Physical volume "/dev/mapper/mpatha1" successfully created.
  device-mapper: reload ioctl on  (253:27) failed: Device or resource busy
  Failed to activate new LV.
  Failed to create sanlock lv lvmlock in vg foo
  Failed to create internal lv.
  Failed to initialize lock args for lock type sanlock
  Volume group "foo" successfully removed

Jul 31 11:15:55 harding-02 kernel: device-mapper: table: 253:27: linear: Device lookup failed
Jul 31 11:15:55 harding-02 kernel: device-mapper: ioctl: error adding target to table

# with a pv created above, this now "works"
[root@harding-02 ~]# vgcreate --shared foo /dev/mapper/mpatha1 
  Enabling sanlock global lock
  Logical volume "lvmlock" created.
  Volume group "foo" successfully created
  VG foo starting sanlock lockspace
  Starting locking.  Waiting until locks are ready...

[root@harding-02 ~]# vgcreate --shared bar /dev/mapper/mpathb1 
  Physical volume "/dev/mapper/mpathb1" successfully created.
  device-mapper: reload ioctl on  (253:28) failed: Device or resource busy
  Failed to activate new LV.
  Failed to create sanlock lv lvmlock in vg bar
  Failed to create internal lv.
  Failed to initialize lock args for lock type sanlock
  Volume group "bar" successfully removed

[root@harding-02 ~]# vgremove foo
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:27) failed: Device or resource busy
  Unable to deactivate foo-lvmlock (253:27).
  Failed to deactivate sanlock lv foo/lvmlock
  Volume group "foo" successfully removed

[root@harding-02 ~]# dmsetup ls | grep foo
foo-lvmlock     (253:27)


Version-Release number of selected component (if applicable):
3.10.0-927.el7.x86_64

lvm2-2.02.180-1.el7    BUILT: Fri Jul 20 12:21:35 CDT 2018
lvm2-libs-2.02.180-1.el7    BUILT: Fri Jul 20 12:21:35 CDT 2018
lvm2-cluster-2.02.180-1.el7    BUILT: Fri Jul 20 12:21:35 CDT 2018
lvm2-lockd-2.02.180-1.el7    BUILT: Fri Jul 20 12:21:35 CDT 2018
lvm2-python-boom-0.9-4.el7    BUILT: Fri Jul 20 12:23:30 CDT 2018
cmirror-2.02.180-1.el7    BUILT: Fri Jul 20 12:21:35 CDT 2018
device-mapper-1.02.149-1.el7    BUILT: Fri Jul 20 12:21:35 CDT 2018
device-mapper-libs-1.02.149-1.el7    BUILT: Fri Jul 20 12:21:35 CDT 2018
device-mapper-event-1.02.149-1.el7    BUILT: Fri Jul 20 12:21:35 CDT 2018
device-mapper-event-libs-1.02.149-1.el7    BUILT: Fri Jul 20 12:21:35 CDT 2018
device-mapper-persistent-data-0.7.3-3.el7    BUILT: Tue Nov 14 05:07:18 CST 2017
sanlock-3.6.0-1.el7    BUILT: Tue Dec  5 11:47:21 CST 2017
sanlock-lib-3.6.0-1.el7    BUILT: Tue Dec  5 11:47:21 CST 2017


How reproducible:
Everytime

Comment 3 Corey Marthaler 2018-07-31 18:13:05 UTC

This does not appear to exist in the original 7.6 lvm build.

3.10.0-927.el7.x86_64
lvm2-2.02.179-1.el7    BUILT: Mon Jun 18 01:12:41 CDT 2018
lvm2-libs-2.02.179-1.el7    BUILT: Mon Jun 18 01:12:41 CDT 2018
lvm2-cluster-2.02.179-1.el7    BUILT: Mon Jun 18 01:12:41 CDT 2018
lvm2-lockd-2.02.179-1.el7    BUILT: Mon Jun 18 01:12:41 CDT 2018
lvm2-python-boom-0.9-4.el7    BUILT: Fri Jul 20 12:23:30 CDT 2018
cmirror-2.02.179-1.el7    BUILT: Mon Jun 18 01:12:41 CDT 2018
device-mapper-1.02.148-1.el7    BUILT: Mon Jun 18 01:12:41 CDT 2018
device-mapper-libs-1.02.148-1.el7    BUILT: Mon Jun 18 01:12:41 CDT 2018
device-mapper-event-1.02.148-1.el7    BUILT: Mon Jun 18 01:12:41 CDT 2018
device-mapper-event-libs-1.02.148-1.el7    BUILT: Mon Jun 18 01:12:41 CDT 2018
device-mapper-persistent-data-0.7.3-3.el7    BUILT: Tue Nov 14 05:07:18 CST 2017
sanlock-3.6.0-1.el7    BUILT: Tue Dec  5 11:47:21 CST 2017
sanlock-lib-3.6.0-1.el7    BUILT: Tue Dec  5 11:47:21 CST 2017


[root@harding-02 ~]# vgcreate --shared foo /dev/mapper/mpathc1
  Enabling sanlock global lock
  Physical volume "/dev/mapper/mpathc1" successfully created.
  Logical volume "lvmlock" created.
  Volume group "foo" successfully created
  VG foo starting sanlock lockspace
  Starting locking.  Waiting until locks are ready...
[root@harding-02 ~]# vgcreate --shared bar /dev/mapper/mpathb1
  Physical volume "/dev/mapper/mpathb1" successfully created.
  Logical volume "lvmlock" created.
  Volume group "bar" successfully created
  VG bar starting sanlock lockspace
  Starting locking.  Waiting until locks are ready...
[root@harding-02 ~]# vgremove -f bar foo
  Volume group "bar" successfully removed
  Volume group "foo" successfully removed

Comment 4 David Teigland 2018-07-31 18:22:40 UTC

I get the same vgcreate issue.  It's probably udev, related to the patch I had to add to close and reopen devices RDWR for other udev problems.

Comment 5 Corey Marthaler 2018-07-31 18:26:08 UTC

Appears to have gone into 179-2.

[root@harding-02 ~]# vgcreate --shared foo /dev/mapper/mpathc1
  Enabling sanlock global lock
  Physical volume "/dev/mapper/mpathc1" successfully created.
  device-mapper: reload ioctl on  (253:27) failed: Device or resource busy
  Failed to activate new LV.
  Failed to create sanlock lv lvmlock in vg foo
  Failed to create internal lv.
  Failed to initialize lock args for lock type sanlock
  Volume group "foo" successfully removed
[root@harding-02 ~]# /usr/tests/sts-rhel7.6/lvm2/bin/lvm_rpms 
3.10.0-927.el7.x86_64

lvm2-2.02.179-2.el7    BUILT: Thu Jun 21 08:02:34 CDT 2018
lvm2-libs-2.02.179-2.el7    BUILT: Thu Jun 21 08:02:34 CDT 2018
lvm2-cluster-2.02.179-2.el7    BUILT: Thu Jun 21 08:02:34 CDT 2018
lvm2-lockd-2.02.179-2.el7    BUILT: Thu Jun 21 08:02:34 CDT 2018
lvm2-python-boom-0.9-4.el7    BUILT: Fri Jul 20 12:23:30 CDT 2018
cmirror-2.02.179-2.el7    BUILT: Thu Jun 21 08:02:34 CDT 2018
device-mapper-1.02.148-2.el7    BUILT: Thu Jun 21 08:02:34 CDT 2018
device-mapper-libs-1.02.148-2.el7    BUILT: Thu Jun 21 08:02:34 CDT 2018
device-mapper-event-1.02.148-2.el7    BUILT: Thu Jun 21 08:02:34 CDT 2018
device-mapper-event-libs-1.02.148-2.el7    BUILT: Thu Jun 21 08:02:34 CDT 2018
device-mapper-persistent-data-0.7.3-3.el7    BUILT: Tue Nov 14 05:07:18 CST 2017
sanlock-3.6.0-1.el7    BUILT: Tue Dec  5 11:47:21 CST 2017
sanlock-lib-3.6.0-1.el7    BUILT: Tue Dec  5 11:47:21 CST 2017

Comment 6 David Teigland 2018-07-31 19:27:17 UTC

#libdm-deptree.c:1944      Creating san-lvmlock
#ioctl/libdm-iface.c:1857          dm create san-lvmlock LVM-113hOfVPegeoqHGO5L5HXRwBK5iEFEvhmUK3XergOoxNKZYrZEURdlQII9GiIylz [ noopencount flush ]   [16384] (*1)
#libdm-deptree.c:2696      Loading table for san-lvmlock (253:10).
#libdm-deptree.c:2641          Adding target to (253:10): 0 524288 linear 7:5 2048
#ioctl/libdm-iface.c:1857          dm table   (253:10) [ opencount flush ]   [16384] (*1)
#ioctl/libdm-iface.c:1857          dm reload   (253:10) [ noopencount flush ]   [16384] (*1)
#ioctl/libdm-iface.c:1895    device-mapper: reload ioctl on  (253:10) failed: Device or resource busy
#libdm-deptree.c:993       Removing san-lvmlock (253:10)
#libdm-common.c:2433          Udev cookie 0xd4d1cf8 (semid 2359296) created
#libdm-common.c:2453          Udev cookie 0xd4d1cf8 (semid 2359296) incremented to 1
#libdm-common.c:2325          Udev cookie 0xd4d1cf8 (semid 2359296) incremented to 2
#libdm-common.c:2575          Udev cookie 0xd4d1cf8 (semid 2359296) assigned to REMOVE task(2) with flags DISABLE_LIBRARY_FALLBACK SUBSYSTEM_0        (0x120)
#ioctl/libdm-iface.c:1857          dm remove   (253:10) [ noopencount flush ]   [16384] (*1)
#libdm-common.c:1487          san-lvmlock: Stacking NODE_DEL [trust_udev]

Comment 7 David Teigland 2018-07-31 20:08:52 UTC

Adding a retry for the reload ioctl doesn't help, it just retries forever:

libdm-deptree.c:1944      Creating san-lvmlock
#ioctl/libdm-iface.c:1857          dm create san-lvmlock LVM-3qVzxeOXMayMnCSdvWkeGTKLlL4c5bcCe9YJTN9xLiGOKXIac43d3itVY0jx6hfc [ noopencount flush ]   [16384] (*1)
#ioctl/libdm-iface.c:1859          ioctl begin command 3241737475 type create san-lvmlock
#ioctl/libdm-iface.c:1861          ioctl end
#libdm-deptree.c:2696      Loading table for san-lvmlock (253:10).
#libdm-deptree.c:2641          Adding target to (253:10): 0 524288 linear 7:5 2048
#ioctl/libdm-iface.c:1857          dm table   (253:10) [ opencount flush ]   [16384] (*1)
#ioctl/libdm-iface.c:1859          ioctl begin command 3241737484 type table
#ioctl/libdm-iface.c:1861          ioctl end
#ioctl/libdm-iface.c:1857          dm reload   (253:10) [ noopencount flush ]   [16384] (*1)
#ioctl/libdm-iface.c:1859          ioctl begin command 3241737481 type reload
#ioctl/libdm-iface.c:1861          ioctl end
#ioctl/libdm-iface.c:1897    device-mapper: reload ioctl on  (253:10) failed: Device or resource busy
#ioctl/libdm-iface.c:2008          retrying reload
#ioctl/libdm-iface.c:1857          dm reload   (253:10) [ noopencount flush ]   [16384] (*1)
#ioctl/libdm-iface.c:1859          ioctl begin command 3241737481 type reload
#ioctl/libdm-iface.c:1861          ioctl end
#ioctl/libdm-iface.c:1897    device-mapper: reload ioctl on  (253:10) failed: Device or resource busy
#ioctl/libdm-iface.c:2008          retrying reload
#ioctl/libdm-iface.c:1857          dm reload   (253:10) [ noopencount flush ]   [16384] (*1)
#ioctl/libdm-iface.c:1859          ioctl begin command 3241737481 type reload
#ioctl/libdm-iface.c:1861          ioctl end

The dm info during the retries:

# dmsetup info san-lvmlock
Name:              san-lvmlock
State:             ACTIVE
Read Ahead:        256
Tables present:    None
Open count:        0
Event number:      0
Major, minor:      253, 10
Number of targets: 0
UUID: LVM-3qVzxeOXMayMnCSdvWkeGTKLlL4c5bcCe9YJTN9xLiGOKXIac43d3itVY0jx6hfc

kernel errors from dmesg:

[1741226.839794] device-mapper: table: 253:10: linear: Device lookup failed
[1741226.847612] device-mapper: ioctl: error adding target to table

Comment 8 David Teigland 2018-07-31 21:38:13 UTC

This diff invalidating/closing bcache for the PVs, prior to creating the sanlock LV, fixes this problem, but I don't know why, so I'm not sure it's the right solution.

diff --git a/lib/locking/lvmlockd.c b/lib/locking/lvmlockd.c
index 24a46e792f87..57089023a816 100644
--- a/lib/locking/lvmlockd.c
+++ b/lib/locking/lvmlockd.c
@@ -591,6 +591,7 @@ static int _init_vg_sanlock(struct cmd_context *cmd, struct volume_group *vg, in
        const char *reply_str;
        const char *vg_lock_args = NULL;
        const char *opts = NULL;
+       struct pv_list *pvl;
        int extend_mb;
        int result;
        int ret;
@@ -600,6 +601,9 @@ static int _init_vg_sanlock(struct cmd_context *cmd, struct volume_group *vg, in
        if (!_lvmlockd_connected)
                return 0;
 
+       dm_list_iterate_items(pvl, &vg->pvs)
+               label_scan_invalidate(pvl->pv->dev);
+
        /*
         * Automatic extension of the sanlock lv is disabled by
         * setting sanlock_lv_extend to 0.  Zero won't work as

Comment 9 David Teigland 2018-08-01 15:33:18 UTC

fixed here
https://sourceware.org/git/?p=lvm2.git;a=commit;h=a75eb8d74c367b88d5b323d5dfb2e6556f2ad680

vgcreate: close exclusive fd after pvcreate

When vgcreate does an automatic pvcreate, it opens the
dev with O_EXCL to ensure no other subsystem is using
the device.  This exclusive fd remained in bcache and
prevented activation parts of lvm from using the dev.

This appeared with vgcreate of a sanlock VG because of
the unique combination where the dev is not yet a PV,
so pvcreate is needed, and the vgcreate also creates
and activates an internal LV for sanlock.

Fix this by closing the exclusive fd after it's used
by pvcreate so that it won't interfere with other
bits of lvm that may try to use the device.

Comment 10 Marian Csontos 2018-08-03 08:50:17 UTC

Corey, this has a TestBlocker set - is not a workaround to create a PVS first working? This should unblock testing (or the scratch build I posted yesterday - https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=17544505)

Comment 11 Corey Marthaler 2018-08-06 21:52:26 UTC

Well technically due to the global lock chicken and egg, an initial pvcreate will fail, hence why our tests do the initial shared VG create in this manner.

[root@harding-02 ~]# pvcreate /dev/mapper/mpatha1
  Global lock failed: check that global lockspace is started

Now, we could add a hack to override lockd, but then we're not testing an "official" setup i believe.

[root@harding-02 ~]# pvcreate --config global/use_lvmlockd=0 /dev/mapper/mpatha1
  Physical volume "/dev/mapper/mpatha1" successfully created.
[root@harding-02 ~]# vgcreate --shared global /dev/mapper/mpatha1
  Enabling sanlock global lock
  Logical volume "lvmlock" created.
  Volume group "global" successfully created
  VG global starting sanlock lockspace
  Starting locking.  Waiting until locks are ready...

I'll attempt your build and then remove the TB flag...

Comment 12 Corey Marthaler 2018-08-06 21:54:30 UTC

Not to mention, even with the passing initial create hack above, we'll still run into the " device-mapper: remove ioctl on" errors later in testing

Comment 14 Corey Marthaler 2018-08-09 23:00:36 UTC

Is it possible this fix only narrowed the window for reproduction? I just saw this similar remove failure with the fixed rpms. I hadn't seen any other issues while testing since installing them however.

3.10.0-931.el7.x86_64
lvm2-2.02.180-2.el7.bz1610455    BUILT: Thu Aug  2 03:47:52 CDT 2018
lvm2-libs-2.02.180-2.el7.bz1610455    BUILT: Thu Aug  2 03:47:52 CDT 2018
lvm2-cluster-2.02.180-2.el7.bz1610455    BUILT: Thu Aug  2 03:47:52 CDT 2018
lvm2-lockd-2.02.180-2.el7.bz1610455    BUILT: Thu Aug  2 03:47:52 CDT 2018
lvm2-python-boom-0.9-5.el7.bz1610455    BUILT: Thu Aug  2 03:47:52 CDT 2018
cmirror-2.02.180-2.el7.bz1610455    BUILT: Thu Aug  2 03:47:52 CDT 2018
device-mapper-1.02.149-2.el7.bz1610455    BUILT: Thu Aug  2 03:47:52 CDT 2018
device-mapper-libs-1.02.149-2.el7.bz1610455    BUILT: Thu Aug  2 03:47:52 CDT 2018
device-mapper-event-1.02.149-2.el7.bz1610455    BUILT: Thu Aug  2 03:47:52 CDT 2018
device-mapper-event-libs-1.02.149-2.el7.bz1610455    BUILT: Thu Aug  2 03:47:52 CDT 2018
device-mapper-persistent-data-0.7.3-3.el7    BUILT: Tue Nov 14 05:07:18 CST 2017
sanlock-3.6.0-1.el7    BUILT: Tue Dec  5 11:47:21 CST 2017
sanlock-lib-3.6.0-1.el7    BUILT: Tue Dec  5 11:47:21 CST 2017



harding-02: pvcreate /dev/mapper/mpatha1 /dev/mapper/mpathb2 /dev/mapper/mpathb1 /dev/mapper/mpathc2 /dev/mapper/mpathc1 /dev/mapper/mpathd2 /dev/mapper/mpathd1 /dev/mapper/mpathe2 /dev/mapper/mpathe1 /dev/mapper/mpathf2
harding-02: vgcreate  --shared raid_sanity /dev/mapper/mpatha1 /dev/mapper/mpathb2 /dev/mapper/mpathb1 /dev/mapper/mpathc2 /dev/mapper/mpathc1 /dev/mapper/mpathd2 /dev/mapper/mpathd1 /dev/mapper/mpathe2 /dev/mapper/mpathe1 /dev/mapper/mpathf2
harding-02: vgchange --lock-start raid_sanity
harding-03: vgchange --lock-start raid_sanity

============================================================
Iteration 1 of 1 started at Thu Aug  9 17:37:19 CDT 2018
============================================================

Recreating PVs/VG with smaller 8G size
harding-03: vgchange --lock-stop raid_sanity

# The assumption here is that the script didn't wait long enough to the lock space to actually be stopped
Aug  9 17:37:22 harding-02 qarshd[64900]: Running cmdline: vgremove raid_sanity
Aug  9 17:37:22 harding-02 lvmlockd[5986]: 1533854242 S lvm_raid_sanity lockspace hosts 1
unable to remove VG raid_sanity


[root@harding-03 ~]# sanlock gets
s lvm_global:1439:/dev/mapper/global-lvmlock:0 

# this should fail since it no longer holds the lock
[root@harding-03 ~]# vgremove raid_sanity
  Cannot access VG raid_sanity due to failed lock.


# Back to harding-02 the intended vgremove node in the first place

[root@harding-02 lvm]# sanlock gets
s lvm_raid_sanity:723:/dev/mapper/raid_sanity-lvmlock:0 
s lvm_global:723:/dev/mapper/global-lvmlock:0 
[root@harding-02 lvm]# vgremove -f raid_sanity
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:28) failed: Device or resource busy
  Unable to deactivate raid_sanity-lvmlock (253:28).
  Failed to deactivate sanlock lv raid_sanity/lvmlock
  Volume group "raid_sanity" successfully removed

Comment 15 David Teigland 2018-08-10 14:07:17 UTC

I think the issue here is that the force option doesn't work properly when the sanlock VG lockspace is still running.  Force skips the normal shutdown/removal of the internal sanlock LV, and the in-use LV causes the dm errors.  I expect you will see the sanlock LV still active in the kernel using dmsetup after the VG is removed.  To make 'vgremove -f' work correctly we can't let 'force' skip the shutdown of the internal LV.

Comment 16 Corey Marthaler 2018-08-10 15:39:36 UTC

Yep.

[root@harding-02 ~]# vgs raid_sanity
  Volume group "raid_sanity" not found
  Cannot process volume group raid_sanity

[root@harding-02 ~]# dmsetup ls | grep raid_sanity
raid_sanity-lvmlock     (253:28)

Comment 18 Corey Marthaler 2018-08-22 19:46:45 UTC

Like mentioned in comment #15 and comment #16, this isn't fully fixed yet, correct? Or is the 'vgremove -f' portion a separate bug?

3.10.0-931.el7.x86_64

lvm2-2.02.180-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
lvm2-libs-2.02.180-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
lvm2-cluster-2.02.180-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
lvm2-lockd-2.02.180-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
lvm2-python-boom-0.9-8.el7    BUILT: Tue Aug 21 11:28:32 CDT 2018
cmirror-2.02.180-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
device-mapper-1.02.149-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
device-mapper-libs-1.02.149-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
device-mapper-event-1.02.149-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
device-mapper-event-libs-1.02.149-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
device-mapper-persistent-data-0.7.3-3.el7    BUILT: Tue Nov 14 05:07:18 CST 2017
sanlock-3.6.0-1.el7    BUILT: Tue Dec  5 11:47:21 CST 2017
sanlock-lib-3.6.0-1.el7    BUILT: Tue Dec  5 11:47:21 CST 2017




[root@harding-03 tmp]# pvscan
  Skipping global lock: lockspace not found or started
  PV /dev/sda2             VG rhel_harding-03   lvm2 [<92.16 GiB / 0    free]
  PV /dev/sdb1             VG rhel_harding-03   lvm2 [<93.16 GiB / 0    free]
  PV /dev/sdc1             VG rhel_harding-03   lvm2 [<93.16 GiB / 0    free]
  Reading VG global without a lock.
  PV /dev/mapper/mpatha1   VG global            lvm2 [249.96 GiB / 249.71 GiB free]
  PV /dev/mapper/mpathd1                        lvm2 [<250.00 GiB]
  PV /dev/mapper/mpathc1                        lvm2 [<250.00 GiB]
  PV /dev/mapper/mpathe1                        lvm2 [<250.00 GiB]
  PV /dev/mapper/mpathb1                        lvm2 [<250.00 GiB]
  Total: 8 [1.49 TiB] / in use: 4 [528.43 GiB] / in no VG: 4 [<1000.00 GiB]

[root@harding-03 tmp]# vgchange --lock-start global
  Skipping global lock: lockspace not found or started
  VG global starting sanlock lockspace
  Starting locking.  Waiting for sanlock may take 20 sec to 3 min...

[root@harding-03 tmp]# vgremove -f global
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  device-mapper: remove ioctl on  (253:19) failed: Device or resource busy
  Unable to deactivate global-lvmlock (253:19).
  Failed to deactivate sanlock lv global/lvmlock
  Volume group "global" successfully removed

[root@harding-03 tmp]# vgs global
  Volume group "global" not found
  Cannot process volume group global

[root@harding-03 tmp]# dmsetup ls | grep lvmlock
global-lvmlock  (253:19)

Comment 19 David Teigland 2018-08-22 19:50:20 UTC

The original bug is fixed (vgcreate works).  vgremove -f is not fixed and is a separate issue, so it should probably be a separate bug.

Comment 20 Corey Marthaler 2018-08-23 16:15:27 UTC

Filed bug 1621491 for the issue mentioned in comment #18.

Comment 21 Corey Marthaler 2018-08-23 18:11:59 UTC

Fix for the vg creation failing verified in the latest rpms.

3.10.0-931.el7.x86_64

lvm2-2.02.180-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
lvm2-libs-2.02.180-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
lvm2-cluster-2.02.180-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
lvm2-lockd-2.02.180-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
lvm2-python-boom-0.9-8.el7    BUILT: Tue Aug 21 11:28:32 CDT 2018
cmirror-2.02.180-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
device-mapper-1.02.149-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
device-mapper-libs-1.02.149-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
device-mapper-event-1.02.149-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
device-mapper-event-libs-1.02.149-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
device-mapper-persistent-data-0.7.3-3.el7    BUILT: Tue Nov 14 05:07:18 CST 2017
sanlock-3.6.0-1.el7    BUILT: Tue Dec  5 11:47:21 CST 2017
sanlock-lib-3.6.0-1.el7    BUILT: Tue Dec  5 11:47:21 CST 2017

Comment 23 errata-xmlrpc 2018-10-30 11:03:51 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3193

Note You need to log in before you can comment on or make changes to this bug.