1758326 – "Assertion failed: can't _pv_write non-orphan PV (in VG )" after attemping to remove sanlock VG that had been converted to local

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1758326 - "Assertion failed: can't _pv_write non-orphan PV (in VG )" after attemping to remove sanlock VG that had been converted to local

Summary: "Assertion failed: can't _pv_write non-orphan PV (in VG )" after attemping to...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	lvm2
Sub Component:
Version:	8.1
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	medium
Target Milestone:	rc
Target Release:	8.0
Assignee:	LVM and device-mapper development team
QA Contact:	cluster-qe@redhat.com
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-10-03 20:40 UTC by Corey Marthaler
Modified:	2021-09-07 11:54 UTC (History)
CC List:	8 users (show)
Fixed In Version:	lvm2-2.03.06-1.el8
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-04-28 16:58:59 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)
verbose vgremove attempt (84.76 KB, text/plain) 2019-10-03 20:50 UTC, Corey Marthaler	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHELPLAN-31461	0	None	None	None	2021-09-07 11:54:05 UTC
Red Hat Product Errata	RHEA-2020:1881	0	None	None	None	2020-04-28 16:59:15 UTC

Description Corey Marthaler 2019-10-03 20:40:29 UTC

Description of problem:
vgcreate  --shared global /dev/sdd1
  Skipping global lock: lockspace not found or started
host-087: vgchange --lock-start global
host-093: vgchange --lock-start global
  Skipping global lock: lockspace not found or started

creating lvm devices...
host-087: pvcreate  /dev/sda1 /dev/sdh1 /dev/sdf1 /dev/sde1 /dev/sdb1
host-087: vgcreate  --shared snapper_thinp /dev/sda1 /dev/sdh1 /dev/sdf1 /dev/sde1 /dev/sdb1
host-087: vgchange --lock-start snapper_thinp
host-093: vgchange --lock-start snapper_thinp

============================================================
Iteration 1 of 1 started at Thu Oct  3 15:32:03 CDT 2019
============================================================
SCENARIO - [convert_thin_snap_to_local]
Convert a shared VG containing thin pool volumes to a local VG
Making pool volume
lvcreate --activate ey --thinpool POOL -L 1G  --zero n --poolmetadatasize 4M snapper_thinp

Skipping meta check until supported with shared storage (bug 1265768|1686539)

Making origin volume
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n origin
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n other1
  WARNING: Sum of all thin volume sizes (2.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other2
  WARNING: Sum of all thin volume sizes (3.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other3
  WARNING: Sum of all thin volume sizes (4.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other4
  WARNING: Sum of all thin volume sizes (5.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other5
  WARNING: Sum of all thin volume sizes (6.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey -y -k n -s /dev/snapper_thinp/origin -n to_local
host-093: vgchange --lock-stop snapper_thinp
verifying sanlock lock stoppage of snapper_thinp on host-093
vgchange -an snapper_thinp
vgchange --lock-stop snapper_thinp
vgchange --yes --lock-type none --lock-opt force snapper_thinp
vgchange -ay snapper_thinp

Removing snap volume snapper_thinp/to_local
lvremove -f /dev/snapper_thinp/to_local
Removing thin origin and other virtual thin volumes
Removing pool snapper_thinp/POOL
Restoring VG back to default parameters
host-093: vgchange --lock-stop snapper_thinp
verifying sanlock lock stoppage of snapper_thinp on host-093
vgremove --yes snapper_thinp
  Assertion failed: can't _pv_write non-orphan PV (in VG )
  Failed to remove physical volume "/dev/sda1" from volume group "snapper_thinp"
  Volume group "snapper_thinp" not properly removed


Version-Release number of selected component (if applicable):
4.18.0-147.el8.x86_64

kernel-4.18.0-147.el8    BUILT: Thu Sep 26 11:15:44 CDT 2019
lvm2-2.03.05-5.el8    BUILT: Thu Sep 26 01:40:57 CDT 2019
lvm2-libs-2.03.05-5.el8    BUILT: Thu Sep 26 01:40:57 CDT 2019
lvm2-dbusd-2.03.05-5.el8    BUILT: Thu Sep 26 01:43:33 CDT 2019
lvm2-lockd-2.03.05-5.el8    BUILT: Thu Sep 26 01:40:57 CDT 2019
device-mapper-1.02.163-5.el8    BUILT: Thu Sep 26 01:40:57 CDT 2019
device-mapper-libs-1.02.163-5.el8    BUILT: Thu Sep 26 01:40:57 CDT 2019
device-mapper-event-1.02.163-5.el8    BUILT: Thu Sep 26 01:40:57 CDT 2019
device-mapper-event-libs-1.02.163-5.el8    BUILT: Thu Sep 26 01:40:57 CDT 2019
device-mapper-persistent-data-0.8.5-2.el8    BUILT: Wed Jun  5 10:28:04 CDT 2019
sanlock-3.8.0-2.el8    BUILT: Wed Jun 12 15:50:27 CDT 2019
sanlock-lib-3.8.0-2.el8    BUILT: Wed Jun 12 15:50:27 CDT 2019


How reproducible:
Everytime

Comment 1 Corey Marthaler 2019-10-03 20:50:50 UTC

Created attachment 1622427 [details]
verbose vgremove attempt

Comment 2 David Teigland 2019-10-04 17:08:56 UTC

When you forcibly change the VG to lock_type none, the internal lvmlock LV is left behind in what is now a non-lockd VG.
You then run vgremove which finds the lvmlock LV is still using one of the PVs, and it prints the error.
A workaround is to simply run 'lvremove vg/lvmlock' before vgremove.  A fix is pushed to master:

https://sourceware.org/git/?p=lvm2.git;a=commitdiff;h=b6240c91882a98df9eceab0312eec893b7051765

Comment 5 Juraj Hrdlica 2020-03-09 17:27:43 UTC

Verified with latest RPMs

lvm2-2.03.08-2.el8.x86_64


 SCENARIO - [convert_thin_snap_to_local]
 Convert a shared VG containing thin pool volumes to a local VG
 Making pool volume
 lvcreate --activate ey --thinpool POOL -L 1G --profile thin-performance --zero n --poolmetadatasize 4M snapper_thinp
 
 Skipping meta check until supported with shared storage (bug 1265768|1686539)
 
 Making origin volume
 lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n origin
 lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other1
   WARNING: Sum of all thin volume sizes (2.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
 lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other2
   WARNING: Sum of all thin volume sizes (3.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
 lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n other3
   WARNING: Sum of all thin volume sizes (4.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
 lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other4
   WARNING: Sum of all thin volume sizes (5.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
 lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n other5
   WARNING: Sum of all thin volume sizes (6.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
 lvcreate --activate ey -y -k n -s /dev/snapper_thinp/origin -n to_local
 virt-148: vgchange --lock-stop snapper_thinp
 Unit sanlock.service could not be found.
 vgchange -an snapper_thinp
 vgchange --lock-stop snapper_thinp
 vgchange --yes --lock-type none --lock-opt force snapper_thinp
 vgchange -ay snapper_thinp
 
 Removing snap volume snapper_thinp/to_local
 lvremove -f /dev/snapper_thinp/to_local
 Removing thin origin and other virtual thin volumes
 Removing pool snapper_thinp/POOL
 Restoring VG back to default parameters
 virt-148: vgchange --lock-stop snapper_thinp
 Unit sanlock.service could not be found.
 vgremove --yes snapper_thinp
 pvremove --yes /dev/sda1 /dev/sda2 /dev/sdb1 /dev/sdb2 /dev/sdc2
 pvcreate /dev/sda1 /dev/sda2 /dev/sdb1 /dev/sdb2 /dev/sdc2
 Checking if leftover devfs entry for vg snapper_thinp is present (BZ 1544409)
 vgcreate --shared snapper_thinp /dev/sda1 /dev/sda2 /dev/sdb1 /dev/sdb2 /dev/sdc2
 virt-147: vgchange --lock-start snapper_thinp
 virt-148: vgchange --lock-start snapper_thinp

Comment 6 Corey Marthaler 2020-03-09 19:45:49 UTC

Verification of this with the existing sts tools requires that you setup sanlock by hand. It appears (based on the 'Unit sanlock.service could not be found.' above) that you ran this with with --env cluster-lvmlockd which doesn't use sanlock. 


kernel-4.18.0-184.el8    BUILT: Tue Feb 25 21:37:02 CST 2020

lvm2-2.03.08-2.el8    BUILT: Mon Feb 24 11:21:38 CST 2020
lvm2-libs-2.03.08-2.el8    BUILT: Mon Feb 24 11:21:38 CST 2020
device-mapper-1.02.169-2.el8    BUILT: Mon Feb 24 11:21:38 CST 2020
device-mapper-libs-1.02.169-2.el8    BUILT: Mon Feb 24 11:21:38 CST 2020
device-mapper-event-1.02.169-2.el8    BUILT: Mon Feb 24 11:21:38 CST 2020
device-mapper-event-libs-1.02.169-2.el8    BUILT: Mon Feb 24 11:21:38 CST 2020

sanlock-3.8.0-2.el8    BUILT: Wed Jun 12 15:50:27 CDT 2019
sanlock-lib-3.8.0-2.el8    BUILT: Wed Jun 12 15:50:27 CDT 2019


I did verify in a sanlock env that this is fixed now:

[root@host-092 sts-rhel8.2]# systemctl status sanlock
        sanlock.service - Shared Storage Lease Manager
   Loaded: loaded (/usr/lib/systemd/system/sanlock.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2020-03-09 11:01:14 CDT; 3h 34min ago
  Process: 856804 ExecStart=/usr/sbin/sanlock daemon (code=exited, status=0/SUCCESS)
 Main PID: 856805 (sanlock)
    Tasks: 6
   Memory: 20.5M
   CGroup: /system.slice/sanlock.service
           ââ856805 /usr/sbin/sanlock daemon
           ââ856806 /usr/sbin/sanlock daemon

Mar 09 11:01:14 host-092.virt.lab.msp.redhat.com systemd[1]: Starting Shared Storage Lease Manager...
Mar 09 11:01:14 host-092.virt.lab.msp.redhat.com systemd[1]: Started Shared Storage Lease Manager.
[root@host-092 sts-rhel8.2]# systemctl status lvmlock
Unit lvmlock.service could not be found.
[root@host-092 sts-rhel8.2]# systemctl status lvmlocks
â lvmlocks.service - LVM locking start and stop
   Loaded: loaded (/usr/lib/systemd/system/lvmlocks.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:lvmlockd(8)
[root@host-092 sts-rhel8.2]# systemctl status lvmlockd
         lvmlockd.service - LVM lock daemon
   Loaded: loaded (/usr/lib/systemd/system/lvmlockd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2020-03-09 11:01:14 CDT; 3h 34min ago
     Docs: man:lvmlockd(8)
 Main PID: 856816 (lvmlockd)
    Tasks: 3
   Memory: 3.2M
   CGroup: /system.slice/lvmlockd.service
           ââ856816 /usr/sbin/lvmlockd --foreground

Mar 09 11:01:14 host-092.virt.lab.msp.redhat.com systemd[1]: Starting LVM lock daemon...
Mar 09 11:01:14 host-092.virt.lab.msp.redhat.com lvmlockd[856816]: [D] creating /run/lvm/lvmlockd.socket
Mar 09 11:01:14 host-092.virt.lab.msp.redhat.com lvmlockd[856816]: 1583769674 lvmlockd started
Mar 09 11:01:14 host-092.virt.lab.msp.redhat.com systemd[1]: Started LVM lock daemon.


host-073: pvcreate  /dev/sde1 /dev/sdh1 /dev/sdg1 /dev/sdd1 /dev/sdf1
host-073: vgcreate  --shared snapper_thinp /dev/sde1 /dev/sdh1 /dev/sdg1 /dev/sdd1 /dev/sdf1
host-073: vgchange --lock-start snapper_thinp
host-083: vgchange --lock-start snapper_thinp

============================================================
Iteration 1 of 1 started at Mon Mar  9 11:06:16 CDT 2020
============================================================
SCENARIO - [convert_thin_snap_to_local]
Convert a shared VG containing thin pool volumes to a local VG
Making pool volume
lvcreate --activate ey --thinpool POOL -L 1G --profile thin-performance --zero y --poolmetadatasize 4M snapper_thinp

Skipping meta check until supported with shared storage (bug 1265768|1686539)

Making origin volume
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n origin
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n other1
  WARNING: Sum of all thin volume sizes (2.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n other2
  WARNING: Sum of all thin volume sizes (3.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other3
  WARNING: Sum of all thin volume sizes (4.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n other4
  WARNING: Sum of all thin volume sizes (5.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other5
  WARNING: Sum of all thin volume sizes (6.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey -y -k n -s /dev/snapper_thinp/origin -n to_local
host-073: vgchange --lock-stop snapper_thinp
verifying sanlock lock stoppage of snapper_thinp on host-073
vgchange -an snapper_thinp
vgchange --lock-stop snapper_thinp
vgchange --yes --lock-type none --lock-opt force snapper_thinp
vgchange -ay snapper_thinp

Removing snap volume snapper_thinp/to_local
lvremove -f /dev/snapper_thinp/to_local
Removing thin origin and other virtual thin volumes
Removing pool snapper_thinp/POOL
Restoring VG back to default parameters
host-073: vgchange --lock-stop snapper_thinp
verifying sanlock lock stoppage of snapper_thinp on host-073
vgremove --yes snapper_thinp
pvremove --yes /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1
pvcreate /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1
Checking if leftover devfs entry for vg snapper_thinp is present (BZ 1544409)
vgcreate --shared snapper_thinp /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1
host-073: vgchange --lock-start snapper_thinp
host-083: vgchange --lock-start snapper_thinp

Comment 8 errata-xmlrpc 2020-04-28 16:58:59 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:1881

Note You need to log in before you can comment on or make changes to this bug.