RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2148528 - lvm commands fail to detect VGs after vgimport because of an outdated hints file
Summary: lvm commands fail to detect VGs after vgimport because of an outdated hints file
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: lvm2
Version: 9.3
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: David Teigland
QA Contact: cluster-qe
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-11-25 18:01 UTC by Alexandros Panagiotou
Modified: 2023-11-07 11:28 UTC (History)
10 users (show)

Fixed In Version: lvm2-2.03.21-1.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-11-07 08:53:27 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker CLUSTERQE-6685 0 None None None 2023-05-16 00:15:14 UTC
Red Hat Issue Tracker RHELPLAN-140529 0 None None None 2022-11-25 18:21:17 UTC
Red Hat Product Errata RHBA-2023:6633 0 None None None 2023-11-07 08:53:53 UTC

Description Alexandros Panagiotou 2022-11-25 18:01:56 UTC 
Description of problem:
After importing a VG with vgimport on a system with shared storage, the VG is
not detected by lvm commands that use the hints file (e.g. vgs, lvs, vgchange,
lvchange), until the file is either manually deleted or pvscan --cache is 
manually run.

Version-Release number of selected component (if applicable):
Observed on a customer system with lvm2-2.03.14-3.el8.x86_64
Reproduced also with lvm2-2.03.14-6.el8.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Create 2 VMs sharing a disk via multiple paths. 
   I have been testing with the same disk on the hypervisor presented
   twice on each VM - with an xml definition containing a <wwn> section
   and a <shareable/> section, but I expect any kind of shared storage
   will behave the same way.
2. Setup multipath to handle the paths and create a partition on the mpath
   device (e.g. mpatha1)
   (the partition may not be necessary, but I have just been copying the
    customer environment)
3. Make sure that both VMs see the partition (e.g. restart multipathd on both
   or reboot the VMs or run kpartx -a /dev/mapper/mpatha)
4. On the 2nd VM: Run vgs so that the hints file gets updated and devs_hash
   is calculated including all devices.
5. On the 1st VM: create a VG on mpatha1 with some content (e.g. vg01 with a
   single LV lv01). Then deactivate and export the VG.
6. On the 2nd VM: Run vgimport vg01 to import the VG. Then run vgs to see the
   newly imported VG.

Actual results:
vgimport succeeds on the 2nd VM, but the newly imported VG is not visible in 
vgs (as well as lvs, vgchange and lvchange)

Deleting /run/lvm/hints or running pvscan --cache makes the VG visible.

Expected results:
The customer was expecting that vgimport would make the VG visible and that was
my initial expectation too. After reviewing lib/label/hints.c I am not so sure
what my expectation should be, as vgimport is listed in the commands that
ignore hints. At a first glance it is counter-intuitive that pvscan --cache is
necessary along with vgimport (it seems like a redundant step).

Additional info:
It is not clear to me if there is a real problem or if this is a matter of
confusion that can be solved with clearer documentation. Potential outcomes
I can think of include:
[1] Make it clear (in a KCS?) that after moving PVs to a system it is good to
    run:

    pvscan --cache

    in order to detect new PVs, especially when these are on shared storage

[2] Modify vgimport to invalidate the hints file so that any next command
    recreates it.

[3] When trying to reproduce the same steps without multipath (on a shared
    scsi disk with a partition) I noticed that systemd was automatically
    running pvscan --cache in the background and the problem didn't reproduce.
    It is not clear (yet?) to me why an mpath device and a scsi disk behave
    differently in this case, but maybe it would make sense to have both behave
    in the same way.
Comment 2 Alexandros Panagiotou 2022-11-25 19:15:51 UTC 
Hello,
As far as I can tell from the tests I have been running, the reason triggering the failure to detect the newly imported VG is that the devices present on the system are not changing - only their content changes. As a result, devs_hash does not change and the check in the end of _read_hint_file:

  if (read_hash && (read_hash != calc_hash)) {

results in accepting the hints file as valid. This can be seen in verbose vgs output with a line similar to:

15:54:32.664444 vgs[1806] label/hints.c:888  accept hints found 1


If the hints file is not updated with all devices (e.g. if step 4 from the description is omitted) then lvm detects the change and this is visible by a message like:

19:40:15.696011 vgs[1832] label/hints.c:883  ignore hints with read_hash
879261312 count 12 calc_hash 2365211306 count 13


In the case of using a shared scsi disk without multipath, the difference is bigger. If we enable debug logging of lvm commands in the journal, we can see systemd initiating a pvscan --cache in the background, right after vgimport, which invalidates the hints file. Then vgs recreates it.

Thanks,
Alexandros
Comment 3 David Teigland 2022-11-28 18:01:42 UTC 
In short, they should disable hints, but could also probably make vgimport invalidate the hints so vgs will see the VG.

A broader point of context, is that vgimport/vgexport cannot fully and properly handle inter-machine transfer of VGs on shared disks.  The customer may be attempting their own form of HA or failover (or using a third party product, which we've also seen doing this), and this is a clue that their process isn't quite right.  When there are shared disks, a couple key recommendations are:  1) enable system IDs on all the hosts and all the VGs, 2) disable hints in lvm.conf on all the hosts (hints="none").  The hints file is a local-disk-only optimization.  One simple way to see the problem with hints and shared disks is to run pvcreate on one host and then run pvs on the other host.  The new PV will not appear on the second host until the hints file is removed or manually refreshed.  If the customer wants to continue using hints, then they will need to find all the places themselves where manual refreshing (or removal) of hints is required in their process.

With all that said, we do our best to minimize the problems with hints and shared disks (even though perfection is not possible.)  So, invalidating the local hints file from vgimport as you suggest seems to make sense.  vgimport isn't a frequent command, so invalidating hints shouldn't noticeably hurt the optimization.  We could use this bz to make that addition.

Also, the intended behaviors should not depend on pvscan running.  It may be useful to disable the udev-generated pvscans when analyzing vgimport/vgexport which don't inherently involve uevents.  Whether pvscans from udev are expected or extraneous (a known udev problem), the pvscan is not a predictable or necessary part of handling hints.  A generated pvscan --cache <dev> does invalidate hints, but this is not actually an essential part of handling hints; it is instead a "backup" invalidation in case there is an unknown case that the devs_hash fails to catch.
Comment 4 David Teigland 2022-11-28 18:14:23 UTC 
Another critical step when using LVM on shared disks is to disable the "autoactivation" setting on the VG with vgchange --setautoactivation n (another old way of doing similar is using autoactivation_volume_list in lvm.conf.)  This will help avoid having multiple hosts activate LVs in the VG simultaneously.  The LVM-activate resource agent is what RH uses in it's HA product for handling VGs on shared disks, so it's a reference for the kind of things that should be considered https://github.com/ClusterLabs/resource-agents/blob/main/heartbeat/LVM-activate
Comment 5 Alexandros Panagiotou 2022-12-21 09:31:30 UTC 
Hello David,
First of all thank you for reviewing and all the suggestions. I have shared this with the customer. Eventually, the customer intends to use these LUNs as part of a Red Hat cluster. I guess the whole vgexport/vgimport process was not necessary in the first place - all that was needed, as far as I see it, would be to make sure autoactivation is disabled for the shared VG (via either the --setautoactivation n or via the auto_activation_volume_list) and then just run pvscan --cache on the 2nd node (or delete the hints file) to discover the newly created shared VG.

In any case, having vgimport invalidate hints sounds like a reasonable thing to me (since vgimport very likely means adding a new PV and it also is not expected to be happening all the time). I also guess that this is more of an RFE at this point, rather than anything else.

Thanks,
Alexandros
Comment 7 David Teigland 2023-02-22 22:36:38 UTC 
Moving this to RHEL9 where the change will be applied first.  It's uncertain if this would qualify for backporting to RHEL8.

pushed change to main branch which invalidates hints from vgimport:
https://sourceware.org/git/?p=lvm2.git;a=commit;h=25cb6a042fab1af7ea4f4a164771a810e1acdef5

This commit tries to reduce potential confusion from unusual vgimport usage shown above.  There is not an actual lvm bug being fixed here, so there may need to be additional changes outside of lvm to address a larger problem in how lvm is being used.
Comment 8 David Teigland 2023-02-22 22:41:41 UTC 
To test this:
- run vgexport vg
- look at /run/lvm/hints
- run vgimport vg
- run vgs
- look at /run/lvm/hints which should be newly created by the vgs command
Comment 10 Corey Marthaler 2023-06-07 22:24:33 UTC 
Marking Verified:Tested in the latest rpms.

kernel-5.14.0-322.el9    BUILT: Fri Jun  2 10:00:53 AM CEST 2023
lvm2-2.03.21-1.el9    BUILT: Fri Apr 21 02:33:33 PM CEST 2023
lvm2-libs-2.03.21-1.el9    BUILT: Fri Apr 21 02:33:33 PM CEST 2023


[root@virt-494 ~]# vgcreate vg /dev/sd[bcd]
  Volume group "vg" successfully created
[root@virt-494 ~]# vgs
  VG            #PV #LV #SN Attr   VSize    VFree   
  vg              3   0   0 wz--n- <224.99g <224.99g

[root@virt-494 ~]# ls -lrt /run/lvm/hints
-rw-------. 1 root root 451 Jun  7 23:01 /run/lvm/hints
[root@virt-494 ~]# cat /run/lvm/hints
# Created by vgs pid 140487 Wed Jun  7 23:01:20 2023
hints_version: 2.1
global_filter:a|.*|
filter:a|.*|
scan_lvs:0
devices_file:system.devices
scan:/dev/vda2 pvid:SsceSlYNjS8kmeqh3LoB3ZLOYO0YPmTz devn:252:2 vg:rhel_virt-494
scan:/dev/sdb pvid:Ib7HTfkrycAwdb5yRt6EQbH8njLZgaQy devn:8:16 vg:vg
scan:/dev/sdc pvid:N3LPQ8g3KPBscB2azDFL3qa1KfouAEuV devn:8:32 vg:vg
scan:/dev/sdd pvid:0mH30CNUjAob72JxjKQnmlZ2lI3g4FHp devn:8:48 vg:vg
devs_hash: 37594913 5



[root@virt-494 ~]# vgexport vg
  Volume group "vg" successfully exported
[root@virt-494 ~]# vgs
  VG            #PV #LV #SN Attr   VSize    VFree   
  vg              3   0   0 wzx-n- <224.99g <224.99g

[root@virt-494 ~]# ls -lrt /run/lvm/hints
-rw-------. 1 root root 451 Jun  7 23:01 /run/lvm/hints
[root@virt-494 ~]# cat /run/lvm/hints
# Created by vgs pid 140487 Wed Jun  7 23:01:20 2023
hints_version: 2.1
global_filter:a|.*|
filter:a|.*|
scan_lvs:0
devices_file:system.devices
scan:/dev/vda2 pvid:SsceSlYNjS8kmeqh3LoB3ZLOYO0YPmTz devn:252:2 vg:rhel_virt-494
scan:/dev/sdb pvid:Ib7HTfkrycAwdb5yRt6EQbH8njLZgaQy devn:8:16 vg:vg
scan:/dev/sdc pvid:N3LPQ8g3KPBscB2azDFL3qa1KfouAEuV devn:8:32 vg:vg
scan:/dev/sdd pvid:0mH30CNUjAob72JxjKQnmlZ2lI3g4FHp devn:8:48 vg:vg
devs_hash: 37594913 5



[root@virt-494 ~]# vgimport vg
  Volume group "vg" successfully imported
[root@virt-494 ~]# vgs
  VG            #PV #LV #SN Attr   VSize    VFree   
  vg              3   0   0 wz--n- <224.99g <224.99g

[root@virt-494 ~]# ls -lrt /run/lvm/hints
-rw-------. 1 root root 451 Jun  7 23:05 /run/lvm/hints
[root@virt-494 ~]# cat /run/lvm/hints
# Created by vgs pid 140647 Wed Jun  7 23:05:33 2023
hints_version: 2.1
global_filter:a|.*|
filter:a|.*|
scan_lvs:0
devices_file:system.devices
scan:/dev/vda2 pvid:SsceSlYNjS8kmeqh3LoB3ZLOYO0YPmTz devn:252:2 vg:rhel_virt-494
scan:/dev/sdb pvid:Ib7HTfkrycAwdb5yRt6EQbH8njLZgaQy devn:8:16 vg:vg
scan:/dev/sdc pvid:N3LPQ8g3KPBscB2azDFL3qa1KfouAEuV devn:8:32 vg:vg
scan:/dev/sdd pvid:0mH30CNUjAob72JxjKQnmlZ2lI3g4FHp devn:8:48 vg:vg
devs_hash: 37594913 5
Comment 15 Corey Marthaler 2023-06-29 18:01:05 UTC 
Marking VERIFIED in the latest rpms as well.

kernel-5.14.0-322.el9    BUILT: Fri Jun  2 10:00:53 AM CEST 2023
lvm2-2.03.21-2.el9    BUILT: Thu May 25 12:03:04 AM CEST 2023
lvm2-libs-2.03.21-2.el9    BUILT: Thu May 25 12:03:04 AM CEST 2023



[root@grant-01 ~]# vgcreate vg /dev/sd[bcd]1
  Volume group "vg" successfully created
[root@grant-01 ~]# cat /run/lvm/hints
# Created empty by vgcreate pid 1158204 Thu Jun 29 19:55:46 2023

[root@grant-01 ~]# pvscan --cache
  pvscan[1158209] PV /dev/sda2 online.
  pvscan[1158209] PV /dev/sdb1 online.
  pvscan[1158209] PV /dev/sdc1 online.
  pvscan[1158209] PV /dev/sdd1 online.
  pvscan[1158209] PV /dev/sde1 not used.
  pvscan[1158209] PV /dev/sdf1 not used.
  pvscan[1158209] PV /dev/sdg1 not used.
[root@grant-01 ~]# cat /run/lvm/hints
# Created by pvscan pid 1158209 Thu Jun 29 19:56:00 2023
hints_version: 2.1
global_filter:a|.*|
filter:a|.*|
scan_lvs:0
devices_file:system.devices
scan:/dev/sda2 pvid:WRlCEBxaRZWLBaIgnWhaQyAa8Uu4u2f8 devn:8:2 vg:rhel_grant-01
scan:/dev/sdb1 pvid:P0zHKVg6VyjsyybVscIsSjQ57QQOjZde devn:8:17 vg:vg
scan:/dev/sdc1 pvid:V5nehGfxQKqylHNOydLsm4lcCaBvlaj3 devn:8:33 vg:vg
scan:/dev/sdd1 pvid:ueNNLv8Ifh3mEGxrBIdcmhNBZtMeqiF3 devn:8:49 vg:vg
scan:/dev/sde1 pvid:njEZWMeq2a8yhSmHgJc79XeBe4iyzY1l devn:8:65 vg:-
scan:/dev/sdf1 pvid:m01NogKn21BQEUWNMivndGAD0re1AneJ devn:8:81 vg:-
scan:/dev/sdg1 pvid:rRoTdlEwuBDzxzLt3gSUmYEMoOzhhUhY devn:8:97 vg:-
devs_hash: 1094215824 10

[root@grant-01 ~]# vgexport vg
  Volume group "vg" successfully exported
[root@grant-01 ~]# cat /run/lvm/hints
# Created by pvscan pid 1158209 Thu Jun 29 19:56:00 2023
hints_version: 2.1
global_filter:a|.*|
filter:a|.*|
scan_lvs:0
devices_file:system.devices
scan:/dev/sda2 pvid:WRlCEBxaRZWLBaIgnWhaQyAa8Uu4u2f8 devn:8:2 vg:rhel_grant-01
scan:/dev/sdb1 pvid:P0zHKVg6VyjsyybVscIsSjQ57QQOjZde devn:8:17 vg:vg
scan:/dev/sdc1 pvid:V5nehGfxQKqylHNOydLsm4lcCaBvlaj3 devn:8:33 vg:vg
scan:/dev/sdd1 pvid:ueNNLv8Ifh3mEGxrBIdcmhNBZtMeqiF3 devn:8:49 vg:vg
scan:/dev/sde1 pvid:njEZWMeq2a8yhSmHgJc79XeBe4iyzY1l devn:8:65 vg:-
scan:/dev/sdf1 pvid:m01NogKn21BQEUWNMivndGAD0re1AneJ devn:8:81 vg:-
scan:/dev/sdg1 pvid:rRoTdlEwuBDzxzLt3gSUmYEMoOzhhUhY devn:8:97 vg:-
devs_hash: 1094215824 10

[root@grant-01 ~]# vgimport vg
  Volume group "vg" successfully imported
[root@grant-01 ~]# cat /run/lvm/hints
# Created by pvscan pid 1158209 Thu Jun 29 19:56:00 2023
hints_version: 2.1
global_filter:a|.*|
filter:a|.*|
scan_lvs:0
devices_file:system.devices
scan:/dev/sda2 pvid:WRlCEBxaRZWLBaIgnWhaQyAa8Uu4u2f8 devn:8:2 vg:rhel_grant-01
scan:/dev/sdb1 pvid:P0zHKVg6VyjsyybVscIsSjQ57QQOjZde devn:8:17 vg:vg
scan:/dev/sdc1 pvid:V5nehGfxQKqylHNOydLsm4lcCaBvlaj3 devn:8:33 vg:vg
scan:/dev/sdd1 pvid:ueNNLv8Ifh3mEGxrBIdcmhNBZtMeqiF3 devn:8:49 vg:vg
scan:/dev/sde1 pvid:njEZWMeq2a8yhSmHgJc79XeBe4iyzY1l devn:8:65 vg:-
scan:/dev/sdf1 pvid:m01NogKn21BQEUWNMivndGAD0re1AneJ devn:8:81 vg:-
scan:/dev/sdg1 pvid:rRoTdlEwuBDzxzLt3gSUmYEMoOzhhUhY devn:8:97 vg:-
devs_hash: 1094215824 10

[root@grant-01 ~]# vgs
  VG            #PV #LV #SN Attr   VSize    VFree 
  rhel_grant-01   1   3   0 wz--n- <446.13g     0 
  vg              3   0   0 wz--n-   <1.31t <1.31t
[root@grant-01 ~]# cat /run/lvm/hints
# Created by vgs pid 1158221 Thu Jun 29 19:57:24 2023
hints_version: 2.1
global_filter:a|.*|
filter:a|.*|
scan_lvs:0
devices_file:system.devices
scan:/dev/sda2 pvid:WRlCEBxaRZWLBaIgnWhaQyAa8Uu4u2f8 devn:8:2 vg:rhel_grant-01
scan:/dev/sdb1 pvid:P0zHKVg6VyjsyybVscIsSjQ57QQOjZde devn:8:17 vg:vg
scan:/dev/sdc1 pvid:V5nehGfxQKqylHNOydLsm4lcCaBvlaj3 devn:8:33 vg:vg
scan:/dev/sdd1 pvid:ueNNLv8Ifh3mEGxrBIdcmhNBZtMeqiF3 devn:8:49 vg:vg
scan:/dev/sde1 pvid:njEZWMeq2a8yhSmHgJc79XeBe4iyzY1l devn:8:65 vg:-
scan:/dev/sdf1 pvid:m01NogKn21BQEUWNMivndGAD0re1AneJ devn:8:81 vg:-
scan:/dev/sdg1 pvid:rRoTdlEwuBDzxzLt3gSUmYEMoOzhhUhY devn:8:97 vg:-
devs_hash: 1094215824 10
Comment 18 errata-xmlrpc 2023-11-07 08:53:27 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6633
Note You need to log in before you can comment on or make changes to this bug.