Bug 1261648 - 'VG metadata comparison' issues after shared VG creation
'VG metadata comparison' issues after shared VG creation
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: lvm2 (Show other bugs)
7.2
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: David Teigland
cluster-qe@redhat.com
: TestBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-09 16:47 EDT by Corey Marthaler
Modified: 2015-11-19 07:48 EST (History)
7 users (show)

See Also:
Fixed In Version: lvm2-2.02.130-2.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-19 07:48:01 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Corey Marthaler 2015-09-09 16:47:07 EDT
Description of problem:
I'm still trying to fine a reliable reproducer for this, I've only seen this a couple times so far.

[root@harding-03 ~]# pvscan
  Skipping global lock: lockspace not found or started
  PV /dev/sda2   VG rhel_harding-03   lvm2 [92.67 GiB / 60.00 MiB free]
  PV /dev/sdb1   VG rhel_harding-03   lvm2 [93.16 GiB / 0    free]
  PV /dev/sdc1   VG rhel_harding-03   lvm2 [93.16 GiB / 0    free]
  Total: 3 [278.98 GiB] / in use: 3 [278.98 GiB] / in no VG: 0 [0   ]
[root@harding-03 ~]#  vgcreate --shared harding /dev/mapper/mpath[abcdefgh]1
  WARNING: shared lock type "sanlock" and lvmlockd are Technology Preview.
  For more information on Technology Preview features, visit:
  https://access.redhat.com/support/offerings/techpreview/
  Enabling sanlock global lock
  Physical volume "/dev/mapper/mpatha1" successfully created
  Physical volume "/dev/mapper/mpathb1" successfully created
  Physical volume "/dev/mapper/mpathc1" successfully created
  Physical volume "/dev/mapper/mpathd1" successfully created
  Physical volume "/dev/mapper/mpathe1" successfully created
  Physical volume "/dev/mapper/mpathf1" successfully created
  Physical volume "/dev/mapper/mpathg1" successfully created
  Physical volume "/dev/mapper/mpathh1" successfully created
  Physical volume "/dev/mapper/mpatha1" successfully created
  Physical volume "/dev/mapper/mpathb1" successfully created
  Physical volume "/dev/mapper/mpathc1" successfully created
  Physical volume "/dev/mapper/mpathd1" successfully created
  Physical volume "/dev/mapper/mpathe1" successfully created
  Physical volume "/dev/mapper/mpathf1" successfully created
  Physical volume "/dev/mapper/mpathg1" successfully created
  Physical volume "/dev/mapper/mpathh1" successfully created
  Logical volume "lvmlock" created.
  Physical volume "/dev/mapper/mpatha1" successfully created
  Physical volume "/dev/mapper/mpathb1" successfully created
  Physical volume "/dev/mapper/mpathc1" successfully created
  Physical volume "/dev/mapper/mpathd1" successfully created
  Physical volume "/dev/mapper/mpathe1" successfully created
  Physical volume "/dev/mapper/mpathf1" successfully created
  Physical volume "/dev/mapper/mpathg1" successfully created
  Physical volume "/dev/mapper/mpathh1" successfully created
  Volume group "harding" successfully created
  VG harding starting sanlock lockspace
  Starting locking.  Waiting until locks are ready...


[root@harding-03 ~]#  vgchange --lock-start harding
  Starting locking.  Waiting until locks are ready...


[root@harding-02 ~]# vgchange --lock-start harding
  Skipping global lock: lockspace not found or started
  VG harding starting sanlock lockspace
  Starting locking.  Waiting until locks are ready...

[root@harding-02 ~]# pvscan
  PV /dev/mapper/mpathd1   VG harding           lvm2 [249.99 GiB / 249.74 GiB free]
  PV /dev/mapper/mpathe1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathc1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathb1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathf1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathg1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpatha1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathh1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/sda2             VG rhel_harding-02   lvm2 [92.67 GiB / 60.00 MiB free]
  PV /dev/sdb1             VG rhel_harding-02   lvm2 [93.16 GiB / 0    free]
  PV /dev/sdc1             VG rhel_harding-02   lvm2 [93.16 GiB / 0    free]
  Total: 11 [2.23 TiB] / in use: 11 [2.23 TiB] / in no VG: 0 [0   ]

[root@harding-03 ~]# pvscan
  PV /dev/mapper/mpatha1   VG harding           lvm2 [249.99 GiB / 249.74 GiB free]
  PV /dev/mapper/mpathb1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathc1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathd1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathe1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathf1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathg1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/mapper/mpathh1   VG harding           lvm2 [249.99 GiB / 249.99 GiB free]
  PV /dev/sda2             VG rhel_harding-03   lvm2 [92.67 GiB / 60.00 MiB free]
  PV /dev/sdb1             VG rhel_harding-03   lvm2 [93.16 GiB / 0    free]
  PV /dev/sdc1             VG rhel_harding-03   lvm2 [93.16 GiB / 0    free]
  Total: 11 [2.23 TiB] / in use: 11 [2.23 TiB] / in no VG: 0 [0   ]


[root@harding-03 ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "rhel_harding-03" using metadata type lvm2
  VG metadata comparison failed
  Volume group "harding" not found
  Cannot process volume group harding

[root@harding-02 ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "rhel_harding-02" using metadata type lvm2
  VG metadata comparison failed
  Volume group "harding" not found
  Cannot process volume group harding


[root@harding-02 ~]# pvscan
  VG metadata comparison failed
  PV /dev/sda2   VG rhel_harding-02   lvm2 [92.67 GiB / 60.00 MiB free]
  PV /dev/sdb1   VG rhel_harding-02   lvm2 [93.16 GiB / 0    free]
  PV /dev/sdc1   VG rhel_harding-02   lvm2 [93.16 GiB / 0    free]
  Total: 3 [278.98 GiB] / in use: 3 [278.98 GiB] / in no VG: 0 [0   ]

[root@harding-03 ~]# pvscan
  VG metadata comparison failed
  PV /dev/sda2   VG rhel_harding-03   lvm2 [92.67 GiB / 60.00 MiB free]
  PV /dev/sdb1   VG rhel_harding-03   lvm2 [93.16 GiB / 0    free]
  PV /dev/sdc1   VG rhel_harding-03   lvm2 [93.16 GiB / 0    free]
  Total: 3 [278.98 GiB] / in use: 3 [278.98 GiB] / in no VG: 0 [0   ]


[root@harding-03 ~]# pvremove -ff --config global/use_lvmlockd=0 /dev/mapper/mpath[abcdefgh]1
  VG metadata comparison failed
  WARNING: Volume Group harding is not consistent.
  Internal error: Physical Volume /dev/mapper/mpatha1 has a label, but is neither in a VG nor orphan.
  Internal error: Physical Volume /dev/mapper/mpathb1 has a label, but is neither in a VG nor orphan.
  Internal error: Physical Volume /dev/mapper/mpathc1 has a label, but is neither in a VG nor orphan.
  Internal error: Physical Volume /dev/mapper/mpathd1 has a label, but is neither in a VG nor orphan.
  Internal error: Physical Volume /dev/mapper/mpathe1 has a label, but is neither in a VG nor orphan.
  Internal error: Physical Volume /dev/mapper/mpathf1 has a label, but is neither in a VG nor orphan.
  Internal error: Physical Volume /dev/mapper/mpathg1 has a label, but is neither in a VG nor orphan.
  Internal error: Physical Volume /dev/mapper/mpathh1 has a label, but is neither in a VG nor orphan.



Version-Release number of selected component (if applicable):
3.10.0-306.el7.x86_64

lvm2-2.02.130-1.el7    BUILT: Wed Sep  9 02:44:18 CDT 2015
lvm2-libs-2.02.130-1.el7    BUILT: Wed Sep  9 02:44:18 CDT 2015
lvm2-cluster-2.02.130-1.el7    BUILT: Wed Sep  9 02:44:18 CDT 2015
device-mapper-1.02.107-1.el7    BUILT: Wed Sep  9 02:44:18 CDT 2015
device-mapper-libs-1.02.107-1.el7    BUILT: Wed Sep  9 02:44:18 CDT 2015
device-mapper-event-1.02.107-1.el7    BUILT: Wed Sep  9 02:44:18 CDT 2015
device-mapper-event-libs-1.02.107-1.el7    BUILT: Wed Sep  9 02:44:18 CDT 2015
device-mapper-persistent-data-0.5.5-1.el7    BUILT: Thu Aug 13 09:58:10 CDT 2015
cmirror-2.02.130-1.el7    BUILT: Wed Sep  9 02:44:18 CDT 2015
sanlock-3.2.4-1.el7    BUILT: Fri Jun 19 12:48:49 CDT 2015
sanlock-lib-3.2.4-1.el7    BUILT: Fri Jun 19 12:48:49 CDT 2015
lvm2-lockd-2.02.130-1.el7    BUILT: Wed Sep  9 02:44:18 CDT 2015
Comment 1 Corey Marthaler 2015-09-09 19:17:45 EDT
This seems to always happen after a 'vgscan' command.
Comment 2 Corey Marthaler 2015-09-10 17:57:21 EDT
Adding blocker flag as I can't seem to get beyond this issue for more in depth testing of lvmlockd.

setting up first "global lock" dummy vg for lvmlockd...
harding-02: vgcreate --shared global /dev/mapper/mpathc1
harding-02: vgchange --lock-start global
harding-03: vgchange --lock-start global
  Skipping global lock: lockspace not found or started

creating shared VGs...
Create 2 PV(s) for foo_bar on harding-02
harding-02: pvcreate /dev/mapper/mpathb1 /dev/mapper/mpatha1
harding-02: vgcreate --shared foo_bar /dev/mapper/mpathb1 /dev/mapper/mpatha1
harding-02: vgchange --lock-start foo_bar
harding-03: vgchange --lock-start foo_bar

Create 2 PV(s) for bar_foo on harding-02
harding-02: pvcreate /dev/mapper/mpathd1 /dev/mapper/mpathe1
harding-02: vgcreate --shared bar_foo /dev/mapper/mpathd1 /dev/mapper/mpathe1
harding-02: vgchange --lock-start bar_foo
harding-03: vgchange --lock-start bar_foo

first lvs after this fails and cleanup ends up requiring a reboot.

[root@harding-02 ~]# lvs
  VG metadata comparison failed
  Volume group "bar_foo" not found
  Cannot process volume group bar_foo
  VG metadata comparison failed
  Volume group "foo_bar" not found
  Cannot process volume group foo_bar
  LV   VG              Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home rhel_harding-02 -wi-ao---- 200.97g
  root rhel_harding-02 -wi-ao----  50.00g
  swap rhel_harding-02 -wi-ao----  27.95g
Comment 5 Corey Marthaler 2015-09-23 16:49:58 EDT
Fix verified in the latest rpms.

vgcreate --shared global /dev/sdd2
host-113.virt.lab.msp.redhat.com: vgchange --lock-start global

creating lvm devices...
host-113.virt.lab.msp.redhat.com: pvcreate /dev/sdd1 /dev/sdb2 /dev/sdb1 /dev/sdc2 /dev/sdc1 /dev/sda2 /dev/sda1 /dev/sde2
host-113.virt.lab.msp.redhat.com: vgcreate --shared snapper_thinp /dev/sdd1 /dev/sdb2 /dev/sdb1 /dev/sdc2 /dev/sdc1 /dev/sda2 /dev/sda1 /dev/sde2
host-113.virt.lab.msp.redhat.com: vgchange --lock-start snapper_thinp

lvcreate  --thinpool POOL --profile thin-performance --zero y -L 1G --poolmetadatasize 4M snapper_thinp

Making origin volume
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n origin
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other1
  WARNING: Sum of all thin volume sizes (2.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB)!
lvcreate  -V 1G -T snapper_thinp/POOL -n other2
  WARNING: Sum of all thin volume sizes (3.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB)!
lvcreate  -V 1G -T snapper_thinp/POOL -n other3
  WARNING: Sum of all thin volume sizes (4.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB)!
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other4
  WARNING: Sum of all thin volume sizes (5.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB)!
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other5
  WARNING: Sum of all thin volume sizes (6.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB)!
Making snapshot of origin volume
lvcreate  -k n -s /dev/snapper_thinp/origin -n display_snap


3.10.0-306.el7.x86_64
lvm2-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
lvm2-libs-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
lvm2-cluster-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-libs-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-event-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-event-libs-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-persistent-data-0.5.5-1.el7    BUILT: Thu Aug 13 09:58:10 CDT 2015
cmirror-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
sanlock-3.2.4-1.el7    BUILT: Fri Jun 19 12:48:49 CDT 2015
sanlock-lib-3.2.4-1.el7    BUILT: Fri Jun 19 12:48:49 CDT 2015
lvm2-lockd-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
Comment 6 errata-xmlrpc 2015-11-19 07:48:01 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2147.html

Note You need to log in before you can comment on or make changes to this bug.