Bug 1685723

Summary: exclusive native vdo with lvmlockd will leave remnant lock and cause vgchange to fail "VG stop failed: LVs must first be deactivated"
Product: Red Hat Enterprise Linux 8 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: David Teigland <teigland>
lvm2 sub component: LVM lock daemon / lvmlockd QA Contact: cluster-qe <cluster-qe>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: high CC: agk, heinzm, jbrassow, mcsontos, pasik, prajnoha, teigland, zkabelac
Version: 8.0   
Target Milestone: rc   
Target Release: 8.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: lvm2-2.03.11-0.2.20201103git8801a86.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-18 15:01:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Corey Marthaler 2019-03-05 22:07:02 UTC
Description of problem:
[root@host-083 ~]# pcs status
Cluster name: STSRHTS28976
Stack: corosync
Current DC: host-084 (version 2.0.1-4.el8-0eb7991564) - partition with quorum
Last updated: Tue Mar  5 15:59:08 2019
Last change: Tue Mar  5 15:34:00 2019 by root via cibadmin on host-085

3 nodes configured
9 resources configured

Online: [ host-083 host-084 host-085 ]


[root@host-083 ~]# vgcreate  --shared VG /dev/sdf3 /dev/sdf2 /dev/sdf1 /dev/sdb3 /dev/sdb2 /dev/sdb1 /dev/sda3
  Volume group "VG" successfully created
  VG VG starting dlm lockspace
  Starting locking.  Waiting until locks are ready...

[root@host-084 ~]#  vgchange --lock-start VG
  VG VG starting dlm lockspace
  Starting locking.  Waiting until locks are ready...

[root@host-085 ~]#  vgchange --lock-start VG
  VG VG starting dlm lockspace
  Starting locking.  Waiting until locks are ready...

[root@host-084 ~]# lvcreate --activate ey --yes --type vdo -n vdo1 -L 4G VG
  Logical volume "vdo1" created.
[root@host-084 ~]# lvs -a -o +devices
  LV              VG            Attr       LSize    Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices        
  vdo1            VG            vwi-a-v--- 1016.00m vpool0        0.00                                    vpool0(0)      
  vpool0          VG            dwi-ao----    4.00g               75.05                                   vpool0_vdata(0)
  [vpool0_vdata]  VG            Dwi-ao----    4.00g                                                       /dev/sdg3(0)   

[root@host-084 ~]# lvchange -an VG
[root@host-084 ~]# lvs -a -o +devices
  LV              VG            Attr       LSize    Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices        
  vdo1            VG            vwi---v--- 1016.00m vpool0                                                vpool0(0)      
  vpool0          VG            dwi-------    4.00g                                                       vpool0_vdata(0)
  [vpool0_vdata]  VG            Dwi-------    4.00g                                                       /dev/sdg3(0)   


[root@host-084 ~]# lvremove VG
Removing VDO pool "vpool0" will remove 1 dependent volume(s). Proceed? [y/n]: y
  Logical volume "vdo1" successfully removed

[root@host-084 ~]# lvs -a -o +devices
  LV              VG            Attr       LSize   Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices        
[root@host-084 ~]# dmsetup ls
[root@host-084 ~]# 

[root@host-084 ~]# vgchange --lock-stop VG
  VG VG stop failed: LVs must first be deactivated


[root@host-084 ~]# dlm_tool ls
dlm lockspaces
name          lvm_VG
id            0x0eebdff0
flags         0x00000000 
change        member 3 joined 1 remove 0 failed 0 seq 2,2
members       1 2 3 

name          lvm_global
id            0x12aabd2d
flags         0x00000000 
change        member 3 joined 1 remove 0 failed 0 seq 1,2
members       1 2 3 


[root@host-084 ~]# lvmlockctl -i
VG vg_uuid= lock_type= 
LS  lvm_global
LK VG un ver 0
LK GL un ver 2

VG VG lock_type=dlm TQqIz6-hfGP-G2jg-SARC-c0e5-V0fd-PFrKFl
LS dlm lvm_VG
LK VG un ver 5
LK LV ex OfD74f-tSXQ-Jk6D-oycl-WTJg-gsWo-GYQLSj

### All that lvmdbusd noise actually comes in handy here:
[root@host-084 ~]# grep OfD74f-tSXQ-Jk6D-oycl-WTJg-gsWo-GYQLSj /var/log/messages
Mar  5 15:59:51 host-084 lvmdbusd[3196]: 3196:3196 - LV OfD74f-tSXQ-Jk6D-oycl-WTJg-gsWo-GYQLSj vpool0 with lv_attr dwi-ao----, lv_attr[0] = 'd' is not known
Mar  5 15:59:51 host-084 lvmdbusd[3196]: 3196:3196 - LV OfD74f-tSXQ-Jk6D-oycl-WTJg-gsWo-GYQLSj vpool0 with lv_attr dwi-ao----, lv_attr[0] = 'd' is not known
Mar  5 15:59:52 host-084 lvmdbusd[3196]: 3196:3196 - LV OfD74f-tSXQ-Jk6D-oycl-WTJg-gsWo-GYQLSj vpool0 with lv_attr dwi-ao----, lv_attr[0] = 'd' is not known
Mar  5 15:59:52 host-084 lvmdbusd[3196]: 3196:3196 - LV OfD74f-tSXQ-Jk6D-oycl-WTJg-gsWo-GYQLSj vpool0 with lv_attr dwi-ao----, lv_attr[0] = 'd' is not known
Mar  5 16:00:17 host-084 lvmdbusd[3196]: 3196:3196 - LV OfD74f-tSXQ-Jk6D-oycl-WTJg-gsWo-GYQLSj vpool0 with lv_attr dwi-ao----, lv_attr[0] = 'd' is not known
Mar  5 16:00:17 host-084 lvmdbusd[3196]: 3196:3196 - LV OfD74f-tSXQ-Jk6D-oycl-WTJg-gsWo-GYQLSj vpool0 with lv_attr dwi-------, lv_attr[0] = 'd' is not known



Version-Release number of selected component (if applicable):
kernel-4.18.0-74.el8    BUILT: Wed Feb 27 12:52:17 CST 2019
lvm2-2.03.02-6.el8    BUILT: Fri Feb 22 04:47:54 CST 2019
lvm2-libs-2.03.02-6.el8    BUILT: Fri Feb 22 04:47:54 CST 2019
lvm2-dbusd-2.03.02-6.el8    BUILT: Fri Feb 22 04:50:28 CST 2019
lvm2-lockd-2.03.02-6.el8    BUILT: Fri Feb 22 04:47:54 CST 2019
boom-boot-0.9-7.el8    BUILT: Mon Jan 14 14:00:54 CST 2019
cmirror-2.03.02-6.el8    BUILT: Fri Feb 22 04:47:54 CST 2019
device-mapper-1.02.155-6.el8    BUILT: Fri Feb 22 04:47:54 CST 2019
device-mapper-libs-1.02.155-6.el8    BUILT: Fri Feb 22 04:47:54 CST 2019
device-mapper-event-1.02.155-6.el8    BUILT: Fri Feb 22 04:47:54 CST 2019
device-mapper-event-libs-1.02.155-6.el8    BUILT: Fri Feb 22 04:47:54 CST 2019
device-mapper-persistent-data-0.7.6-1.el8    BUILT: Sun Aug 12 04:21:55 CDT 2018
sanlock-3.6.0-5.el8    BUILT: Thu Dec  6 13:31:26 CST 2018
sanlock-lib-3.6.0-5.el8    BUILT: Thu Dec  6 13:31:26 CST 2018
vdo-6.2.0.293-10.el8    BUILT: Fri Dec 14 18:18:47 CST 2018
kmod-kvdo-6.2.0.293-50.el8    BUILT: Mon Feb 25 16:53:12 CST 2019


How reproducible:
Everytime

Comment 1 David Teigland 2020-09-29 20:05:04 UTC
pushed to master
https://sourceware.org/git/?p=lvm2.git;a=commit;h=2272a32e6f9b99f98514e58de82f8a3baa8b46da

$ lvcreate --type vdo -L5G -n vv test
WARNING: vdo signature detected on /dev/test/vpool0 at offset 0. Wipe it? [y/n]: y
  Wiping vdo signature on /dev/test/vpool0.
    Logical blocks defaulted to 523108 blocks.
    The VDO volume can address 2 GB in 1 data slab.
    It can grow to address at most 16 TB of physical storage in 8192 slabs.
    If a larger maximum size might be needed, use bigger slabs.
  Logical volume "vv" created.

$ lvs -o+uuid test
  LV     VG   Attr       LSize Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert LV UUID                               
  vpool0 test dwi------- 5.00g               60.03                                   fhFoim-Lj9R-Gfbs-fwms-gfVQ-VFCo-ZS4RVr
  vv     test vwi-a-v--- 1.99g vpool0        0.00                                    YdwVMl-dvia-wNyu-Nm2s-d9Ua-lcDe-Uzv81R

$ dlm_tool lockdebug lvm_test

Resource len  4  "VGLK"
Master           
LVB len 32 seq 2
01 01 00 00 49 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Granted
00000001 NL                                    

Resource len 38  "fhFoim-Lj9R-Gfbs-fwms-gfVQ-VFCo-ZS4RVr"
Master           
Granted
00000002 EX                                    

$ lvremove test/vv
Do you really want to remove active logical volume test/vv? [y/n]: y
  Logical volume "vv" successfully removed

$ dlm_tool lockdebug lvm_test

Resource len  4  "VGLK"
Master           
LVB len 32 seq 3
01 01 00 00 4a 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Granted
00000001 NL                                    

Resource len 38  "fhFoim-Lj9R-Gfbs-fwms-gfVQ-VFCo-ZS4RVr"
Master Dir

$ vgchange --lockstop test

$ dlm_tool ls

$ lvmlockctl -i

$ dmsetup ls
rhel-swap       (253:1)
rhel-root       (253:0)

Comment 7 Corey Marthaler 2020-12-16 02:04:11 UTC
Fix verified in the latest rpms.

kernel-4.18.0-259.el8.dt4    BUILT: Sat Dec 12 14:40:07 CST 2020
lvm2-2.03.11-0.3.20201210git9fe7aba.el8    BUILT: Thu Dec 10 09:44:53 CST 2020
lvm2-libs-2.03.11-0.3.20201210git9fe7aba.el8    BUILT: Thu Dec 10 09:44:53 CST 2020


[root@host-087 ~]# vgs
  VG            #PV #LV #SN Attr   VSize    VFree
  testvg          8   0   0 wz--ns <239.91g <239.91g

[root@host-087 ~]# lvcreate --activate ey --yes --type vdo -n vdo1 -L 6G testvg
    Logical blocks defaulted to 523108 blocks.
    The VDO volume can address 2 GB in 1 data slab.
    It can grow to address at most 16 TB of physical storage in 8192 slabs.
    If a larger maximum size might be needed, use bigger slabs.
  Logical volume "vdo1" created.

[root@host-087 ~]# lvs -a -o +devices
  LV              VG            Attr       LSize   Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices
  vdo1            testvg        vwi-a-v---   1.99g vpool0        0.00                                    vpool0(0)
  vpool0          testvg        dwi-------   6.00g               66.69                                   vpool0_vdata(0)
  [vpool0_vdata]  testvg        Dwi-ao----   6.00g                                                       /dev/sda1(0)

[root@host-087 ~]# lvchange -an testvg
[root@host-087 ~]# lvs -a -o +devices
  LV              VG            Attr       LSize   Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices
  vdo1            testvg        vwi---v---   1.99g vpool0                                                vpool0(0)
  vpool0          testvg        dwi-------   6.00g                                                       vpool0_vdata(0)
  [vpool0_vdata]  testvg        Dwi-------   6.00g                                                       /dev/sda1(0)

[root@host-087 ~]# lvremove testvg
Removing VDO pool "vpool0" will remove 1 dependent volume(s). Proceed? [y/n]: y
  Logical volume "vdo1" successfully removed

[root@host-087 ~]# dmsetup ls
[root@host-087 ~]# vgchange --lock-stop testvg
[root@host-087 ~]# dlm_tool ls
[root@host-087 ~]# lvmlockctl -i

Comment 9 errata-xmlrpc 2021-05-18 15:01:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1659