RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1261648 - 'VG metadata comparison' issues after shared VG creation
Summary: 'VG metadata comparison' issues after shared VG creation
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: lvm2
Version: 7.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: David Teigland
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-09-09 20:47 UTC by Corey Marthaler
Modified: 2021-09-03 12:38 UTC (History)
7 users (show)

Fixed In Version: lvm2-2.02.130-2.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-11-19 12:48:01 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2147 0 normal SHIPPED_LIVE lvm2 bug fix and enhancement update 2015-11-19 11:11:07 UTC

Description Corey Marthaler 2015-09-09 20:47:07 UTC
Description of problem:
I'm still trying to fine a reliable reproducer for this, I've only seen this a couple times so far.

[root@harding-03 ~]# pvscan
  Skipping global lock: lockspace not found or started
  PV /dev/sda2   VG rhel_harding-03   lvm2 [92.67 GiB / 60.00 MiB free]
  PV /dev/sdb1   VG rhel_harding-03   lvm2 [93.16 GiB / 0    free]
  PV /dev/sdc1   VG rhel_harding-03   lvm2 [93.16 GiB / 0    free]
  Total: 3 [278.98 GiB] / in use: 3 [278.98 GiB] / in no VG: 0 [0   ]
[root@harding-03 ~]#  vgcreate --shared harding /dev/mapper/mpath[abcdefgh]1
  WARNING: shared lock type "sanlock" and lvmlockd are Technology Preview.
  For more information on Technology Preview features, visit:
  https://access.redhat.com/support/offerings/techpreview/
  Enabling sanlock global lock
  Physical volume "/dev/mapper/mpatha1" successfully created
  Physical volume "/dev/mapper/mpathb1" successfully created
  Physical volume "/dev/mapper/mpathc1" successfully created
  Physical volume "/dev/mapper/mpathd1" successfully created
  Physical volume "/dev/mapper/mpathe1" successfully created
  Physical volume "/dev/mapper/mpathf1" successfully created
  Physical volume "/dev/mapper/mpathg1" successfully created
  Physical volume "/dev/mapper/mpathh1" successfully created
  Physical volume "/dev/mapper/mpatha1" successfully created
  Physical volume "/dev/mapper/mpathb1" successfully created
  Physical volume "/dev/mapper/mpathc1" successfully created
  Physical volume "/dev/mapper/mpathd1" successfully created
  Physical volume "/dev/mapper/mpathe1" successfully created
  Physical volume "/dev/mapper/mpathf1" successfully created
  Physical volume "/dev/mapper/mpathg1" successfully created
  Physical volume "/dev/mapper/mpathh1" successfully created
  Logical volume "lvmlock" created.
  Physical volume "/dev/mapper/mpatha1" successfully created
  Physical volume "/dev/mapper/mpathb1" successfully created
  Physical volume "/dev/mapper/mpathc1" successfully created
  Physical volume "/dev/mapper/mpathd1" successfully created
  Physical volume "/dev/mapper/mpathe1" successfully created
  Physical volume "/dev/mapper/mpathf1" successfully created
  Physical volume "/dev/mapper/mpathg1" successfully created
  Physical volume "/dev/mapper/mpathh1" successfully created
  Volume group "harding" successfully created
  VG harding starting sanlock lockspace
  Starting locking.  Waiting until locks are ready...


[root@harding-03 ~]#  vgchange --lock-start harding
  Starting locking.  Waiting until locks are ready...


[root@harding-02 ~]# vgchange --lock-start harding
  Skipping global lock: lockspace not found or started
  VG harding starting sanlock lockspace
  Starting locking.  Waiting until locks are ready...

[root@harding-02 ~]# pvscan
  PV /dev/mapper/mpathd1   VG harding           lvm2 [249.99 GiB / 249.74 GiB free]
  PV /dev/mapper/mpathe1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathc1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathb1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathf1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathg1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpatha1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathh1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/sda2             VG rhel_harding-02   lvm2 [92.67 GiB / 60.00 MiB free]
  PV /dev/sdb1             VG rhel_harding-02   lvm2 [93.16 GiB / 0    free]
  PV /dev/sdc1             VG rhel_harding-02   lvm2 [93.16 GiB / 0    free]
  Total: 11 [2.23 TiB] / in use: 11 [2.23 TiB] / in no VG: 0 [0   ]

[root@harding-03 ~]# pvscan
  PV /dev/mapper/mpatha1   VG harding           lvm2 [249.99 GiB / 249.74 GiB free]
  PV /dev/mapper/mpathb1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathc1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathd1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathe1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathf1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathg1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathh1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/sda2             VG rhel_harding-03   lvm2 [92.67 GiB / 60.00 MiB free]
  PV /dev/sdb1             VG rhel_harding-03   lvm2 [93.16 GiB / 0    free]
  PV /dev/sdc1             VG rhel_harding-03   lvm2 [93.16 GiB / 0    free]
  Total: 11 [2.23 TiB] / in use: 11 [2.23 TiB] / in no VG: 0 [0   ]


[root@harding-03 ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "rhel_harding-03" using metadata type lvm2
  VG metadata comparison failed
  Volume group "harding" not found
  Cannot process volume group harding

[root@harding-02 ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "rhel_harding-02" using metadata type lvm2
  VG metadata comparison failed
  Volume group "harding" not found
  Cannot process volume group harding


[root@harding-02 ~]# pvscan
  VG metadata comparison failed
  PV /dev/sda2   VG rhel_harding-02   lvm2 [92.67 GiB / 60.00 MiB free]
  PV /dev/sdb1   VG rhel_harding-02   lvm2 [93.16 GiB / 0    free]
  PV /dev/sdc1   VG rhel_harding-02   lvm2 [93.16 GiB / 0    free]
  Total: 3 [278.98 GiB] / in use: 3 [278.98 GiB] / in no VG: 0 [0   ]

[root@harding-03 ~]# pvscan
  VG metadata comparison failed
  PV /dev/sda2   VG rhel_harding-03   lvm2 [92.67 GiB / 60.00 MiB free]
  PV /dev/sdb1   VG rhel_harding-03   lvm2 [93.16 GiB / 0    free]
  PV /dev/sdc1   VG rhel_harding-03   lvm2 [93.16 GiB / 0    free]
  Total: 3 [278.98 GiB] / in use: 3 [278.98 GiB] / in no VG: 0 [0   ]


[root@harding-03 ~]# pvremove -ff --config global/use_lvmlockd=0 /dev/mapper/mpath[abcdefgh]1
  VG metadata comparison failed
  WARNING: Volume Group harding is not consistent.
  Internal error: Physical Volume /dev/mapper/mpatha1 has a label, but is neither in a VG nor orphan.
  Internal error: Physical Volume /dev/mapper/mpathb1 has a label, but is neither in a VG nor orphan.
  Internal error: Physical Volume /dev/mapper/mpathc1 has a label, but is neither in a VG nor orphan.
  Internal error: Physical Volume /dev/mapper/mpathd1 has a label, but is neither in a VG nor orphan.
  Internal error: Physical Volume /dev/mapper/mpathe1 has a label, but is neither in a VG nor orphan.
  Internal error: Physical Volume /dev/mapper/mpathf1 has a label, but is neither in a VG nor orphan.
  Internal error: Physical Volume /dev/mapper/mpathg1 has a label, but is neither in a VG nor orphan.
  Internal error: Physical Volume /dev/mapper/mpathh1 has a label, but is neither in a VG nor orphan.



Version-Release number of selected component (if applicable):
3.10.0-306.el7.x86_64

lvm2-2.02.130-1.el7    BUILT: Wed Sep  9 02:44:18 CDT 2015
lvm2-libs-2.02.130-1.el7    BUILT: Wed Sep  9 02:44:18 CDT 2015
lvm2-cluster-2.02.130-1.el7    BUILT: Wed Sep  9 02:44:18 CDT 2015
device-mapper-1.02.107-1.el7    BUILT: Wed Sep  9 02:44:18 CDT 2015
device-mapper-libs-1.02.107-1.el7    BUILT: Wed Sep  9 02:44:18 CDT 2015
device-mapper-event-1.02.107-1.el7    BUILT: Wed Sep  9 02:44:18 CDT 2015
device-mapper-event-libs-1.02.107-1.el7    BUILT: Wed Sep  9 02:44:18 CDT 2015
device-mapper-persistent-data-0.5.5-1.el7    BUILT: Thu Aug 13 09:58:10 CDT 2015
cmirror-2.02.130-1.el7    BUILT: Wed Sep  9 02:44:18 CDT 2015
sanlock-3.2.4-1.el7    BUILT: Fri Jun 19 12:48:49 CDT 2015
sanlock-lib-3.2.4-1.el7    BUILT: Fri Jun 19 12:48:49 CDT 2015
lvm2-lockd-2.02.130-1.el7    BUILT: Wed Sep  9 02:44:18 CDT 2015

Comment 1 Corey Marthaler 2015-09-09 23:17:45 UTC
This seems to always happen after a 'vgscan' command.

Comment 2 Corey Marthaler 2015-09-10 21:57:21 UTC
Adding blocker flag as I can't seem to get beyond this issue for more in depth testing of lvmlockd.

setting up first "global lock" dummy vg for lvmlockd...
harding-02: vgcreate --shared global /dev/mapper/mpathc1
harding-02: vgchange --lock-start global
harding-03: vgchange --lock-start global
  Skipping global lock: lockspace not found or started

creating shared VGs...
Create 2 PV(s) for foo_bar on harding-02
harding-02: pvcreate /dev/mapper/mpathb1 /dev/mapper/mpatha1
harding-02: vgcreate --shared foo_bar /dev/mapper/mpathb1 /dev/mapper/mpatha1
harding-02: vgchange --lock-start foo_bar
harding-03: vgchange --lock-start foo_bar

Create 2 PV(s) for bar_foo on harding-02
harding-02: pvcreate /dev/mapper/mpathd1 /dev/mapper/mpathe1
harding-02: vgcreate --shared bar_foo /dev/mapper/mpathd1 /dev/mapper/mpathe1
harding-02: vgchange --lock-start bar_foo
harding-03: vgchange --lock-start bar_foo

first lvs after this fails and cleanup ends up requiring a reboot.

[root@harding-02 ~]# lvs
  VG metadata comparison failed
  Volume group "bar_foo" not found
  Cannot process volume group bar_foo
  VG metadata comparison failed
  Volume group "foo_bar" not found
  Cannot process volume group foo_bar
  LV   VG              Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home rhel_harding-02 -wi-ao---- 200.97g
  root rhel_harding-02 -wi-ao----  50.00g
  swap rhel_harding-02 -wi-ao----  27.95g

Comment 5 Corey Marthaler 2015-09-23 20:49:58 UTC
Fix verified in the latest rpms.

vgcreate --shared global /dev/sdd2
host-113.virt.lab.msp.redhat.com: vgchange --lock-start global

creating lvm devices...
host-113.virt.lab.msp.redhat.com: pvcreate /dev/sdd1 /dev/sdb2 /dev/sdb1 /dev/sdc2 /dev/sdc1 /dev/sda2 /dev/sda1 /dev/sde2
host-113.virt.lab.msp.redhat.com: vgcreate --shared snapper_thinp /dev/sdd1 /dev/sdb2 /dev/sdb1 /dev/sdc2 /dev/sdc1 /dev/sda2 /dev/sda1 /dev/sde2
host-113.virt.lab.msp.redhat.com: vgchange --lock-start snapper_thinp

lvcreate  --thinpool POOL --profile thin-performance --zero y -L 1G --poolmetadatasize 4M snapper_thinp

Making origin volume
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n origin
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other1
  WARNING: Sum of all thin volume sizes (2.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB)!
lvcreate  -V 1G -T snapper_thinp/POOL -n other2
  WARNING: Sum of all thin volume sizes (3.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB)!
lvcreate  -V 1G -T snapper_thinp/POOL -n other3
  WARNING: Sum of all thin volume sizes (4.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB)!
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other4
  WARNING: Sum of all thin volume sizes (5.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB)!
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other5
  WARNING: Sum of all thin volume sizes (6.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB)!
Making snapshot of origin volume
lvcreate  -k n -s /dev/snapper_thinp/origin -n display_snap


3.10.0-306.el7.x86_64
lvm2-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
lvm2-libs-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
lvm2-cluster-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-libs-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-event-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-event-libs-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-persistent-data-0.5.5-1.el7    BUILT: Thu Aug 13 09:58:10 CDT 2015
cmirror-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
sanlock-3.2.4-1.el7    BUILT: Fri Jun 19 12:48:49 CDT 2015
sanlock-lib-3.2.4-1.el7    BUILT: Fri Jun 19 12:48:49 CDT 2015
lvm2-lockd-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015

Comment 6 errata-xmlrpc 2015-11-19 12:48:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2147.html


Note You need to log in before you can comment on or make changes to this bug.