Bug 1758326
| Summary: | "Assertion failed: can't _pv_write non-orphan PV (in VG )" after attemping to remove sanlock VG that had been converted to local | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Corey Marthaler <cmarthal> | ||||
| Component: | lvm2 | Assignee: | LVM and device-mapper development team <lvm-team> | ||||
| lvm2 sub component: | LVM lock daemon / lvmlockd | QA Contact: | cluster-qe <cluster-qe> | ||||
| Status: | CLOSED ERRATA | Docs Contact: | |||||
| Severity: | medium | ||||||
| Priority: | unspecified | CC: | agk, heinzm, jbrassow, jhrdlica, mcsontos, prajnoha, teigland, zkabelac | ||||
| Version: | 8.1 | Flags: | pm-rhel:
mirror+
|
||||
| Target Milestone: | rc | ||||||
| Target Release: | 8.0 | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | lvm2-2.03.06-1.el8 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2020-04-28 16:58:59 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Corey Marthaler
2019-10-03 20:40:29 UTC
Created attachment 1622427 [details]
verbose vgremove attempt
When you forcibly change the VG to lock_type none, the internal lvmlock LV is left behind in what is now a non-lockd VG. You then run vgremove which finds the lvmlock LV is still using one of the PVs, and it prints the error. A workaround is to simply run 'lvremove vg/lvmlock' before vgremove. A fix is pushed to master: https://sourceware.org/git/?p=lvm2.git;a=commitdiff;h=b6240c91882a98df9eceab0312eec893b7051765 Verified with latest RPMs lvm2-2.03.08-2.el8.x86_64 SCENARIO - [convert_thin_snap_to_local] Convert a shared VG containing thin pool volumes to a local VG Making pool volume lvcreate --activate ey --thinpool POOL -L 1G --profile thin-performance --zero n --poolmetadatasize 4M snapper_thinp Skipping meta check until supported with shared storage (bug 1265768|1686539) Making origin volume lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n origin lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other1 WARNING: Sum of all thin volume sizes (2.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB). lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other2 WARNING: Sum of all thin volume sizes (3.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB). lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n other3 WARNING: Sum of all thin volume sizes (4.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB). lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other4 WARNING: Sum of all thin volume sizes (5.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB). lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n other5 WARNING: Sum of all thin volume sizes (6.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB). lvcreate --activate ey -y -k n -s /dev/snapper_thinp/origin -n to_local virt-148: vgchange --lock-stop snapper_thinp Unit sanlock.service could not be found. vgchange -an snapper_thinp vgchange --lock-stop snapper_thinp vgchange --yes --lock-type none --lock-opt force snapper_thinp vgchange -ay snapper_thinp Removing snap volume snapper_thinp/to_local lvremove -f /dev/snapper_thinp/to_local Removing thin origin and other virtual thin volumes Removing pool snapper_thinp/POOL Restoring VG back to default parameters virt-148: vgchange --lock-stop snapper_thinp Unit sanlock.service could not be found. vgremove --yes snapper_thinp pvremove --yes /dev/sda1 /dev/sda2 /dev/sdb1 /dev/sdb2 /dev/sdc2 pvcreate /dev/sda1 /dev/sda2 /dev/sdb1 /dev/sdb2 /dev/sdc2 Checking if leftover devfs entry for vg snapper_thinp is present (BZ 1544409) vgcreate --shared snapper_thinp /dev/sda1 /dev/sda2 /dev/sdb1 /dev/sdb2 /dev/sdc2 virt-147: vgchange --lock-start snapper_thinp virt-148: vgchange --lock-start snapper_thinp Verification of this with the existing sts tools requires that you setup sanlock by hand. It appears (based on the 'Unit sanlock.service could not be found.' above) that you ran this with with --env cluster-lvmlockd which doesn't use sanlock.
kernel-4.18.0-184.el8 BUILT: Tue Feb 25 21:37:02 CST 2020
lvm2-2.03.08-2.el8 BUILT: Mon Feb 24 11:21:38 CST 2020
lvm2-libs-2.03.08-2.el8 BUILT: Mon Feb 24 11:21:38 CST 2020
device-mapper-1.02.169-2.el8 BUILT: Mon Feb 24 11:21:38 CST 2020
device-mapper-libs-1.02.169-2.el8 BUILT: Mon Feb 24 11:21:38 CST 2020
device-mapper-event-1.02.169-2.el8 BUILT: Mon Feb 24 11:21:38 CST 2020
device-mapper-event-libs-1.02.169-2.el8 BUILT: Mon Feb 24 11:21:38 CST 2020
sanlock-3.8.0-2.el8 BUILT: Wed Jun 12 15:50:27 CDT 2019
sanlock-lib-3.8.0-2.el8 BUILT: Wed Jun 12 15:50:27 CDT 2019
I did verify in a sanlock env that this is fixed now:
[root@host-092 sts-rhel8.2]# systemctl status sanlock
sanlock.service - Shared Storage Lease Manager
Loaded: loaded (/usr/lib/systemd/system/sanlock.service; disabled; vendor preset: disabled)
Active: active (running) since Mon 2020-03-09 11:01:14 CDT; 3h 34min ago
Process: 856804 ExecStart=/usr/sbin/sanlock daemon (code=exited, status=0/SUCCESS)
Main PID: 856805 (sanlock)
Tasks: 6
Memory: 20.5M
CGroup: /system.slice/sanlock.service
ââ856805 /usr/sbin/sanlock daemon
ââ856806 /usr/sbin/sanlock daemon
Mar 09 11:01:14 host-092.virt.lab.msp.redhat.com systemd[1]: Starting Shared Storage Lease Manager...
Mar 09 11:01:14 host-092.virt.lab.msp.redhat.com systemd[1]: Started Shared Storage Lease Manager.
[root@host-092 sts-rhel8.2]# systemctl status lvmlock
Unit lvmlock.service could not be found.
[root@host-092 sts-rhel8.2]# systemctl status lvmlocks
â lvmlocks.service - LVM locking start and stop
Loaded: loaded (/usr/lib/systemd/system/lvmlocks.service; disabled; vendor preset: disabled)
Active: inactive (dead)
Docs: man:lvmlockd(8)
[root@host-092 sts-rhel8.2]# systemctl status lvmlockd
lvmlockd.service - LVM lock daemon
Loaded: loaded (/usr/lib/systemd/system/lvmlockd.service; disabled; vendor preset: disabled)
Active: active (running) since Mon 2020-03-09 11:01:14 CDT; 3h 34min ago
Docs: man:lvmlockd(8)
Main PID: 856816 (lvmlockd)
Tasks: 3
Memory: 3.2M
CGroup: /system.slice/lvmlockd.service
ââ856816 /usr/sbin/lvmlockd --foreground
Mar 09 11:01:14 host-092.virt.lab.msp.redhat.com systemd[1]: Starting LVM lock daemon...
Mar 09 11:01:14 host-092.virt.lab.msp.redhat.com lvmlockd[856816]: [D] creating /run/lvm/lvmlockd.socket
Mar 09 11:01:14 host-092.virt.lab.msp.redhat.com lvmlockd[856816]: 1583769674 lvmlockd started
Mar 09 11:01:14 host-092.virt.lab.msp.redhat.com systemd[1]: Started LVM lock daemon.
host-073: pvcreate /dev/sde1 /dev/sdh1 /dev/sdg1 /dev/sdd1 /dev/sdf1
host-073: vgcreate --shared snapper_thinp /dev/sde1 /dev/sdh1 /dev/sdg1 /dev/sdd1 /dev/sdf1
host-073: vgchange --lock-start snapper_thinp
host-083: vgchange --lock-start snapper_thinp
============================================================
Iteration 1 of 1 started at Mon Mar 9 11:06:16 CDT 2020
============================================================
SCENARIO - [convert_thin_snap_to_local]
Convert a shared VG containing thin pool volumes to a local VG
Making pool volume
lvcreate --activate ey --thinpool POOL -L 1G --profile thin-performance --zero y --poolmetadatasize 4M snapper_thinp
Skipping meta check until supported with shared storage (bug 1265768|1686539)
Making origin volume
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n origin
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n other1
WARNING: Sum of all thin volume sizes (2.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n other2
WARNING: Sum of all thin volume sizes (3.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other3
WARNING: Sum of all thin volume sizes (4.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n other4
WARNING: Sum of all thin volume sizes (5.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other5
WARNING: Sum of all thin volume sizes (6.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey -y -k n -s /dev/snapper_thinp/origin -n to_local
host-073: vgchange --lock-stop snapper_thinp
verifying sanlock lock stoppage of snapper_thinp on host-073
vgchange -an snapper_thinp
vgchange --lock-stop snapper_thinp
vgchange --yes --lock-type none --lock-opt force snapper_thinp
vgchange -ay snapper_thinp
Removing snap volume snapper_thinp/to_local
lvremove -f /dev/snapper_thinp/to_local
Removing thin origin and other virtual thin volumes
Removing pool snapper_thinp/POOL
Restoring VG back to default parameters
host-073: vgchange --lock-stop snapper_thinp
verifying sanlock lock stoppage of snapper_thinp on host-073
vgremove --yes snapper_thinp
pvremove --yes /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1
pvcreate /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1
Checking if leftover devfs entry for vg snapper_thinp is present (BZ 1544409)
vgcreate --shared snapper_thinp /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1
host-073: vgchange --lock-start snapper_thinp
host-083: vgchange --lock-start snapper_thinp
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:1881 |