RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1685723 - exclusive native vdo with lvmlockd will leave remnant lock and cause vgchange to fail "VG stop failed: LVs must first be deactivated"
Summary: exclusive native vdo with lvmlockd will leave remnant lock and cause vgchange...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: lvm2
Version: 8.0
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: rc
: 8.0
Assignee: David Teigland
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-05 22:07 UTC by Corey Marthaler
Modified: 2021-09-07 11:50 UTC (History)
8 users (show)

Fixed In Version: lvm2-2.03.11-0.2.20201103git8801a86.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-18 15:01:41 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Corey Marthaler 2019-03-05 22:07:02 UTC 
Description of problem:
[root@host-083 ~]# pcs status
Cluster name: STSRHTS28976
Stack: corosync
Current DC: host-084 (version 2.0.1-4.el8-0eb7991564) - partition with quorum
Last updated: Tue Mar  5 15:59:08 2019
Last change: Tue Mar  5 15:34:00 2019 by root via cibadmin on host-085

3 nodes configured
9 resources configured

Online: [ host-083 host-084 host-085 ]


[root@host-083 ~]# vgcreate  --shared VG /dev/sdf3 /dev/sdf2 /dev/sdf1 /dev/sdb3 /dev/sdb2 /dev/sdb1 /dev/sda3
  Volume group "VG" successfully created
  VG VG starting dlm lockspace
  Starting locking.  Waiting until locks are ready...

[root@host-084 ~]#  vgchange --lock-start VG
  VG VG starting dlm lockspace
  Starting locking.  Waiting until locks are ready...

[root@host-085 ~]#  vgchange --lock-start VG
  VG VG starting dlm lockspace
  Starting locking.  Waiting until locks are ready...

[root@host-084 ~]# lvcreate --activate ey --yes --type vdo -n vdo1 -L 4G VG
  Logical volume "vdo1" created.
[root@host-084 ~]# lvs -a -o +devices
  LV              VG            Attr       LSize    Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices        
  vdo1            VG            vwi-a-v--- 1016.00m vpool0        0.00                                    vpool0(0)      
  vpool0          VG            dwi-ao----    4.00g               75.05                                   vpool0_vdata(0)
  [vpool0_vdata]  VG            Dwi-ao----    4.00g                                                       /dev/sdg3(0)   

[root@host-084 ~]# lvchange -an VG
[root@host-084 ~]# lvs -a -o +devices
  LV              VG            Attr       LSize    Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices        
  vdo1            VG            vwi---v--- 1016.00m vpool0                                                vpool0(0)      
  vpool0          VG            dwi-------    4.00g                                                       vpool0_vdata(0)
  [vpool0_vdata]  VG            Dwi-------    4.00g                                                       /dev/sdg3(0)   


[root@host-084 ~]# lvremove VG
Removing VDO pool "vpool0" will remove 1 dependent volume(s). Proceed? [y/n]: y
  Logical volume "vdo1" successfully removed

[root@host-084 ~]# lvs -a -o +devices
  LV              VG            Attr       LSize   Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices        
[root@host-084 ~]# dmsetup ls
[root@host-084 ~]# 

[root@host-084 ~]# vgchange --lock-stop VG
  VG VG stop failed: LVs must first be deactivated


[root@host-084 ~]# dlm_tool ls
dlm lockspaces
name          lvm_VG
id            0x0eebdff0
flags         0x00000000 
change        member 3 joined 1 remove 0 failed 0 seq 2,2
members       1 2 3 

name          lvm_global
id            0x12aabd2d
flags         0x00000000 
change        member 3 joined 1 remove 0 failed 0 seq 1,2
members       1 2 3 


[root@host-084 ~]# lvmlockctl -i
VG vg_uuid= lock_type= 
LS  lvm_global
LK VG un ver 0
LK GL un ver 2

VG VG lock_type=dlm TQqIz6-hfGP-G2jg-SARC-c0e5-V0fd-PFrKFl
LS dlm lvm_VG
LK VG un ver 5
LK LV ex OfD74f-tSXQ-Jk6D-oycl-WTJg-gsWo-GYQLSj

### All that lvmdbusd noise actually comes in handy here:
[root@host-084 ~]# grep OfD74f-tSXQ-Jk6D-oycl-WTJg-gsWo-GYQLSj /var/log/messages
Mar  5 15:59:51 host-084 lvmdbusd[3196]: 3196:3196 - LV OfD74f-tSXQ-Jk6D-oycl-WTJg-gsWo-GYQLSj vpool0 with lv_attr dwi-ao----, lv_attr[0] = 'd' is not known
Mar  5 15:59:51 host-084 lvmdbusd[3196]: 3196:3196 - LV OfD74f-tSXQ-Jk6D-oycl-WTJg-gsWo-GYQLSj vpool0 with lv_attr dwi-ao----, lv_attr[0] = 'd' is not known
Mar  5 15:59:52 host-084 lvmdbusd[3196]: 3196:3196 - LV OfD74f-tSXQ-Jk6D-oycl-WTJg-gsWo-GYQLSj vpool0 with lv_attr dwi-ao----, lv_attr[0] = 'd' is not known
Mar  5 15:59:52 host-084 lvmdbusd[3196]: 3196:3196 - LV OfD74f-tSXQ-Jk6D-oycl-WTJg-gsWo-GYQLSj vpool0 with lv_attr dwi-ao----, lv_attr[0] = 'd' is not known
Mar  5 16:00:17 host-084 lvmdbusd[3196]: 3196:3196 - LV OfD74f-tSXQ-Jk6D-oycl-WTJg-gsWo-GYQLSj vpool0 with lv_attr dwi-ao----, lv_attr[0] = 'd' is not known
Mar  5 16:00:17 host-084 lvmdbusd[3196]: 3196:3196 - LV OfD74f-tSXQ-Jk6D-oycl-WTJg-gsWo-GYQLSj vpool0 with lv_attr dwi-------, lv_attr[0] = 'd' is not known



Version-Release number of selected component (if applicable):
kernel-4.18.0-74.el8    BUILT: Wed Feb 27 12:52:17 CST 2019
lvm2-2.03.02-6.el8    BUILT: Fri Feb 22 04:47:54 CST 2019
lvm2-libs-2.03.02-6.el8    BUILT: Fri Feb 22 04:47:54 CST 2019
lvm2-dbusd-2.03.02-6.el8    BUILT: Fri Feb 22 04:50:28 CST 2019
lvm2-lockd-2.03.02-6.el8    BUILT: Fri Feb 22 04:47:54 CST 2019
boom-boot-0.9-7.el8    BUILT: Mon Jan 14 14:00:54 CST 2019
cmirror-2.03.02-6.el8    BUILT: Fri Feb 22 04:47:54 CST 2019
device-mapper-1.02.155-6.el8    BUILT: Fri Feb 22 04:47:54 CST 2019
device-mapper-libs-1.02.155-6.el8    BUILT: Fri Feb 22 04:47:54 CST 2019
device-mapper-event-1.02.155-6.el8    BUILT: Fri Feb 22 04:47:54 CST 2019
device-mapper-event-libs-1.02.155-6.el8    BUILT: Fri Feb 22 04:47:54 CST 2019
device-mapper-persistent-data-0.7.6-1.el8    BUILT: Sun Aug 12 04:21:55 CDT 2018
sanlock-3.6.0-5.el8    BUILT: Thu Dec  6 13:31:26 CST 2018
sanlock-lib-3.6.0-5.el8    BUILT: Thu Dec  6 13:31:26 CST 2018
vdo-6.2.0.293-10.el8    BUILT: Fri Dec 14 18:18:47 CST 2018
kmod-kvdo-6.2.0.293-50.el8    BUILT: Mon Feb 25 16:53:12 CST 2019


How reproducible:
Everytime
Comment 1 David Teigland 2020-09-29 20:05:04 UTC 
pushed to master
https://sourceware.org/git/?p=lvm2.git;a=commit;h=2272a32e6f9b99f98514e58de82f8a3baa8b46da

$ lvcreate --type vdo -L5G -n vv test
WARNING: vdo signature detected on /dev/test/vpool0 at offset 0. Wipe it? [y/n]: y
  Wiping vdo signature on /dev/test/vpool0.
    Logical blocks defaulted to 523108 blocks.
    The VDO volume can address 2 GB in 1 data slab.
    It can grow to address at most 16 TB of physical storage in 8192 slabs.
    If a larger maximum size might be needed, use bigger slabs.
  Logical volume "vv" created.

$ lvs -o+uuid test
  LV     VG   Attr       LSize Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert LV UUID                               
  vpool0 test dwi------- 5.00g               60.03                                   fhFoim-Lj9R-Gfbs-fwms-gfVQ-VFCo-ZS4RVr
  vv     test vwi-a-v--- 1.99g vpool0        0.00                                    YdwVMl-dvia-wNyu-Nm2s-d9Ua-lcDe-Uzv81R

$ dlm_tool lockdebug lvm_test

Resource len  4  "VGLK"
Master           
LVB len 32 seq 2
01 01 00 00 49 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Granted
00000001 NL                                    

Resource len 38  "fhFoim-Lj9R-Gfbs-fwms-gfVQ-VFCo-ZS4RVr"
Master           
Granted
00000002 EX                                    

$ lvremove test/vv
Do you really want to remove active logical volume test/vv? [y/n]: y
  Logical volume "vv" successfully removed

$ dlm_tool lockdebug lvm_test

Resource len  4  "VGLK"
Master           
LVB len 32 seq 3
01 01 00 00 4a 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Granted
00000001 NL                                    

Resource len 38  "fhFoim-Lj9R-Gfbs-fwms-gfVQ-VFCo-ZS4RVr"
Master Dir

$ vgchange --lockstop test

$ dlm_tool ls

$ lvmlockctl -i

$ dmsetup ls
rhel-swap       (253:1)
rhel-root       (253:0)
Comment 7 Corey Marthaler 2020-12-16 02:04:11 UTC 
Fix verified in the latest rpms.

kernel-4.18.0-259.el8.dt4    BUILT: Sat Dec 12 14:40:07 CST 2020
lvm2-2.03.11-0.3.20201210git9fe7aba.el8    BUILT: Thu Dec 10 09:44:53 CST 2020
lvm2-libs-2.03.11-0.3.20201210git9fe7aba.el8    BUILT: Thu Dec 10 09:44:53 CST 2020


[root@host-087 ~]# vgs
  VG            #PV #LV #SN Attr   VSize    VFree
  testvg          8   0   0 wz--ns <239.91g <239.91g

[root@host-087 ~]# lvcreate --activate ey --yes --type vdo -n vdo1 -L 6G testvg
    Logical blocks defaulted to 523108 blocks.
    The VDO volume can address 2 GB in 1 data slab.
    It can grow to address at most 16 TB of physical storage in 8192 slabs.
    If a larger maximum size might be needed, use bigger slabs.
  Logical volume "vdo1" created.

[root@host-087 ~]# lvs -a -o +devices
  LV              VG            Attr       LSize   Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices
  vdo1            testvg        vwi-a-v---   1.99g vpool0        0.00                                    vpool0(0)
  vpool0          testvg        dwi-------   6.00g               66.69                                   vpool0_vdata(0)
  [vpool0_vdata]  testvg        Dwi-ao----   6.00g                                                       /dev/sda1(0)

[root@host-087 ~]# lvchange -an testvg
[root@host-087 ~]# lvs -a -o +devices
  LV              VG            Attr       LSize   Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices
  vdo1            testvg        vwi---v---   1.99g vpool0                                                vpool0(0)
  vpool0          testvg        dwi-------   6.00g                                                       vpool0_vdata(0)
  [vpool0_vdata]  testvg        Dwi-------   6.00g                                                       /dev/sda1(0)

[root@host-087 ~]# lvremove testvg
Removing VDO pool "vpool0" will remove 1 dependent volume(s). Proceed? [y/n]: y
  Logical volume "vdo1" successfully removed

[root@host-087 ~]# dmsetup ls
[root@host-087 ~]# vgchange --lock-stop testvg
[root@host-087 ~]# dlm_tool ls
[root@host-087 ~]# lvmlockctl -i
Comment 9 errata-xmlrpc 2021-05-18 15:01:41 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1659
Note You need to log in before you can comment on or make changes to this bug.