Bug 2148528
| Summary: | lvm commands fail to detect VGs after vgimport because of an outdated hints file | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Alexandros Panagiotou <apanagio> |
| Component: | lvm2 | Assignee: | David Teigland <teigland> |
| lvm2 sub component: | Command-line tools | QA Contact: | cluster-qe <cluster-qe> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | agk, cmarthal, heinzm, jbrassow, mcsontos, msnitzer, prajnoha, teigland, thornber, zkabelac |
| Version: | 9.3 | Keywords: | Triaged |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | lvm2-2.03.21-1.el9 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-11-07 08:53:27 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Hello,
As far as I can tell from the tests I have been running, the reason triggering the failure to detect the newly imported VG is that the devices present on the system are not changing - only their content changes. As a result, devs_hash does not change and the check in the end of _read_hint_file:
if (read_hash && (read_hash != calc_hash)) {
results in accepting the hints file as valid. This can be seen in verbose vgs output with a line similar to:
15:54:32.664444 vgs[1806] label/hints.c:888 accept hints found 1
If the hints file is not updated with all devices (e.g. if step 4 from the description is omitted) then lvm detects the change and this is visible by a message like:
19:40:15.696011 vgs[1832] label/hints.c:883 ignore hints with read_hash
879261312 count 12 calc_hash 2365211306 count 13
In the case of using a shared scsi disk without multipath, the difference is bigger. If we enable debug logging of lvm commands in the journal, we can see systemd initiating a pvscan --cache in the background, right after vgimport, which invalidates the hints file. Then vgs recreates it.
Thanks,
Alexandros
In short, they should disable hints, but could also probably make vgimport invalidate the hints so vgs will see the VG. A broader point of context, is that vgimport/vgexport cannot fully and properly handle inter-machine transfer of VGs on shared disks. The customer may be attempting their own form of HA or failover (or using a third party product, which we've also seen doing this), and this is a clue that their process isn't quite right. When there are shared disks, a couple key recommendations are: 1) enable system IDs on all the hosts and all the VGs, 2) disable hints in lvm.conf on all the hosts (hints="none"). The hints file is a local-disk-only optimization. One simple way to see the problem with hints and shared disks is to run pvcreate on one host and then run pvs on the other host. The new PV will not appear on the second host until the hints file is removed or manually refreshed. If the customer wants to continue using hints, then they will need to find all the places themselves where manual refreshing (or removal) of hints is required in their process. With all that said, we do our best to minimize the problems with hints and shared disks (even though perfection is not possible.) So, invalidating the local hints file from vgimport as you suggest seems to make sense. vgimport isn't a frequent command, so invalidating hints shouldn't noticeably hurt the optimization. We could use this bz to make that addition. Also, the intended behaviors should not depend on pvscan running. It may be useful to disable the udev-generated pvscans when analyzing vgimport/vgexport which don't inherently involve uevents. Whether pvscans from udev are expected or extraneous (a known udev problem), the pvscan is not a predictable or necessary part of handling hints. A generated pvscan --cache <dev> does invalidate hints, but this is not actually an essential part of handling hints; it is instead a "backup" invalidation in case there is an unknown case that the devs_hash fails to catch. Another critical step when using LVM on shared disks is to disable the "autoactivation" setting on the VG with vgchange --setautoactivation n (another old way of doing similar is using autoactivation_volume_list in lvm.conf.) This will help avoid having multiple hosts activate LVs in the VG simultaneously. The LVM-activate resource agent is what RH uses in it's HA product for handling VGs on shared disks, so it's a reference for the kind of things that should be considered https://github.com/ClusterLabs/resource-agents/blob/main/heartbeat/LVM-activate Hello David, First of all thank you for reviewing and all the suggestions. I have shared this with the customer. Eventually, the customer intends to use these LUNs as part of a Red Hat cluster. I guess the whole vgexport/vgimport process was not necessary in the first place - all that was needed, as far as I see it, would be to make sure autoactivation is disabled for the shared VG (via either the --setautoactivation n or via the auto_activation_volume_list) and then just run pvscan --cache on the 2nd node (or delete the hints file) to discover the newly created shared VG. In any case, having vgimport invalidate hints sounds like a reasonable thing to me (since vgimport very likely means adding a new PV and it also is not expected to be happening all the time). I also guess that this is more of an RFE at this point, rather than anything else. Thanks, Alexandros Moving this to RHEL9 where the change will be applied first. It's uncertain if this would qualify for backporting to RHEL8. pushed change to main branch which invalidates hints from vgimport: https://sourceware.org/git/?p=lvm2.git;a=commit;h=25cb6a042fab1af7ea4f4a164771a810e1acdef5 This commit tries to reduce potential confusion from unusual vgimport usage shown above. There is not an actual lvm bug being fixed here, so there may need to be additional changes outside of lvm to address a larger problem in how lvm is being used. To test this: - run vgexport vg - look at /run/lvm/hints - run vgimport vg - run vgs - look at /run/lvm/hints which should be newly created by the vgs command Marking Verified:Tested in the latest rpms. kernel-5.14.0-322.el9 BUILT: Fri Jun 2 10:00:53 AM CEST 2023 lvm2-2.03.21-1.el9 BUILT: Fri Apr 21 02:33:33 PM CEST 2023 lvm2-libs-2.03.21-1.el9 BUILT: Fri Apr 21 02:33:33 PM CEST 2023 [root@virt-494 ~]# vgcreate vg /dev/sd[bcd] Volume group "vg" successfully created [root@virt-494 ~]# vgs VG #PV #LV #SN Attr VSize VFree vg 3 0 0 wz--n- <224.99g <224.99g [root@virt-494 ~]# ls -lrt /run/lvm/hints -rw-------. 1 root root 451 Jun 7 23:01 /run/lvm/hints [root@virt-494 ~]# cat /run/lvm/hints # Created by vgs pid 140487 Wed Jun 7 23:01:20 2023 hints_version: 2.1 global_filter:a|.*| filter:a|.*| scan_lvs:0 devices_file:system.devices scan:/dev/vda2 pvid:SsceSlYNjS8kmeqh3LoB3ZLOYO0YPmTz devn:252:2 vg:rhel_virt-494 scan:/dev/sdb pvid:Ib7HTfkrycAwdb5yRt6EQbH8njLZgaQy devn:8:16 vg:vg scan:/dev/sdc pvid:N3LPQ8g3KPBscB2azDFL3qa1KfouAEuV devn:8:32 vg:vg scan:/dev/sdd pvid:0mH30CNUjAob72JxjKQnmlZ2lI3g4FHp devn:8:48 vg:vg devs_hash: 37594913 5 [root@virt-494 ~]# vgexport vg Volume group "vg" successfully exported [root@virt-494 ~]# vgs VG #PV #LV #SN Attr VSize VFree vg 3 0 0 wzx-n- <224.99g <224.99g [root@virt-494 ~]# ls -lrt /run/lvm/hints -rw-------. 1 root root 451 Jun 7 23:01 /run/lvm/hints [root@virt-494 ~]# cat /run/lvm/hints # Created by vgs pid 140487 Wed Jun 7 23:01:20 2023 hints_version: 2.1 global_filter:a|.*| filter:a|.*| scan_lvs:0 devices_file:system.devices scan:/dev/vda2 pvid:SsceSlYNjS8kmeqh3LoB3ZLOYO0YPmTz devn:252:2 vg:rhel_virt-494 scan:/dev/sdb pvid:Ib7HTfkrycAwdb5yRt6EQbH8njLZgaQy devn:8:16 vg:vg scan:/dev/sdc pvid:N3LPQ8g3KPBscB2azDFL3qa1KfouAEuV devn:8:32 vg:vg scan:/dev/sdd pvid:0mH30CNUjAob72JxjKQnmlZ2lI3g4FHp devn:8:48 vg:vg devs_hash: 37594913 5 [root@virt-494 ~]# vgimport vg Volume group "vg" successfully imported [root@virt-494 ~]# vgs VG #PV #LV #SN Attr VSize VFree vg 3 0 0 wz--n- <224.99g <224.99g [root@virt-494 ~]# ls -lrt /run/lvm/hints -rw-------. 1 root root 451 Jun 7 23:05 /run/lvm/hints [root@virt-494 ~]# cat /run/lvm/hints # Created by vgs pid 140647 Wed Jun 7 23:05:33 2023 hints_version: 2.1 global_filter:a|.*| filter:a|.*| scan_lvs:0 devices_file:system.devices scan:/dev/vda2 pvid:SsceSlYNjS8kmeqh3LoB3ZLOYO0YPmTz devn:252:2 vg:rhel_virt-494 scan:/dev/sdb pvid:Ib7HTfkrycAwdb5yRt6EQbH8njLZgaQy devn:8:16 vg:vg scan:/dev/sdc pvid:N3LPQ8g3KPBscB2azDFL3qa1KfouAEuV devn:8:32 vg:vg scan:/dev/sdd pvid:0mH30CNUjAob72JxjKQnmlZ2lI3g4FHp devn:8:48 vg:vg devs_hash: 37594913 5 Marking VERIFIED in the latest rpms as well. kernel-5.14.0-322.el9 BUILT: Fri Jun 2 10:00:53 AM CEST 2023 lvm2-2.03.21-2.el9 BUILT: Thu May 25 12:03:04 AM CEST 2023 lvm2-libs-2.03.21-2.el9 BUILT: Thu May 25 12:03:04 AM CEST 2023 [root@grant-01 ~]# vgcreate vg /dev/sd[bcd]1 Volume group "vg" successfully created [root@grant-01 ~]# cat /run/lvm/hints # Created empty by vgcreate pid 1158204 Thu Jun 29 19:55:46 2023 [root@grant-01 ~]# pvscan --cache pvscan[1158209] PV /dev/sda2 online. pvscan[1158209] PV /dev/sdb1 online. pvscan[1158209] PV /dev/sdc1 online. pvscan[1158209] PV /dev/sdd1 online. pvscan[1158209] PV /dev/sde1 not used. pvscan[1158209] PV /dev/sdf1 not used. pvscan[1158209] PV /dev/sdg1 not used. [root@grant-01 ~]# cat /run/lvm/hints # Created by pvscan pid 1158209 Thu Jun 29 19:56:00 2023 hints_version: 2.1 global_filter:a|.*| filter:a|.*| scan_lvs:0 devices_file:system.devices scan:/dev/sda2 pvid:WRlCEBxaRZWLBaIgnWhaQyAa8Uu4u2f8 devn:8:2 vg:rhel_grant-01 scan:/dev/sdb1 pvid:P0zHKVg6VyjsyybVscIsSjQ57QQOjZde devn:8:17 vg:vg scan:/dev/sdc1 pvid:V5nehGfxQKqylHNOydLsm4lcCaBvlaj3 devn:8:33 vg:vg scan:/dev/sdd1 pvid:ueNNLv8Ifh3mEGxrBIdcmhNBZtMeqiF3 devn:8:49 vg:vg scan:/dev/sde1 pvid:njEZWMeq2a8yhSmHgJc79XeBe4iyzY1l devn:8:65 vg:- scan:/dev/sdf1 pvid:m01NogKn21BQEUWNMivndGAD0re1AneJ devn:8:81 vg:- scan:/dev/sdg1 pvid:rRoTdlEwuBDzxzLt3gSUmYEMoOzhhUhY devn:8:97 vg:- devs_hash: 1094215824 10 [root@grant-01 ~]# vgexport vg Volume group "vg" successfully exported [root@grant-01 ~]# cat /run/lvm/hints # Created by pvscan pid 1158209 Thu Jun 29 19:56:00 2023 hints_version: 2.1 global_filter:a|.*| filter:a|.*| scan_lvs:0 devices_file:system.devices scan:/dev/sda2 pvid:WRlCEBxaRZWLBaIgnWhaQyAa8Uu4u2f8 devn:8:2 vg:rhel_grant-01 scan:/dev/sdb1 pvid:P0zHKVg6VyjsyybVscIsSjQ57QQOjZde devn:8:17 vg:vg scan:/dev/sdc1 pvid:V5nehGfxQKqylHNOydLsm4lcCaBvlaj3 devn:8:33 vg:vg scan:/dev/sdd1 pvid:ueNNLv8Ifh3mEGxrBIdcmhNBZtMeqiF3 devn:8:49 vg:vg scan:/dev/sde1 pvid:njEZWMeq2a8yhSmHgJc79XeBe4iyzY1l devn:8:65 vg:- scan:/dev/sdf1 pvid:m01NogKn21BQEUWNMivndGAD0re1AneJ devn:8:81 vg:- scan:/dev/sdg1 pvid:rRoTdlEwuBDzxzLt3gSUmYEMoOzhhUhY devn:8:97 vg:- devs_hash: 1094215824 10 [root@grant-01 ~]# vgimport vg Volume group "vg" successfully imported [root@grant-01 ~]# cat /run/lvm/hints # Created by pvscan pid 1158209 Thu Jun 29 19:56:00 2023 hints_version: 2.1 global_filter:a|.*| filter:a|.*| scan_lvs:0 devices_file:system.devices scan:/dev/sda2 pvid:WRlCEBxaRZWLBaIgnWhaQyAa8Uu4u2f8 devn:8:2 vg:rhel_grant-01 scan:/dev/sdb1 pvid:P0zHKVg6VyjsyybVscIsSjQ57QQOjZde devn:8:17 vg:vg scan:/dev/sdc1 pvid:V5nehGfxQKqylHNOydLsm4lcCaBvlaj3 devn:8:33 vg:vg scan:/dev/sdd1 pvid:ueNNLv8Ifh3mEGxrBIdcmhNBZtMeqiF3 devn:8:49 vg:vg scan:/dev/sde1 pvid:njEZWMeq2a8yhSmHgJc79XeBe4iyzY1l devn:8:65 vg:- scan:/dev/sdf1 pvid:m01NogKn21BQEUWNMivndGAD0re1AneJ devn:8:81 vg:- scan:/dev/sdg1 pvid:rRoTdlEwuBDzxzLt3gSUmYEMoOzhhUhY devn:8:97 vg:- devs_hash: 1094215824 10 [root@grant-01 ~]# vgs VG #PV #LV #SN Attr VSize VFree rhel_grant-01 1 3 0 wz--n- <446.13g 0 vg 3 0 0 wz--n- <1.31t <1.31t [root@grant-01 ~]# cat /run/lvm/hints # Created by vgs pid 1158221 Thu Jun 29 19:57:24 2023 hints_version: 2.1 global_filter:a|.*| filter:a|.*| scan_lvs:0 devices_file:system.devices scan:/dev/sda2 pvid:WRlCEBxaRZWLBaIgnWhaQyAa8Uu4u2f8 devn:8:2 vg:rhel_grant-01 scan:/dev/sdb1 pvid:P0zHKVg6VyjsyybVscIsSjQ57QQOjZde devn:8:17 vg:vg scan:/dev/sdc1 pvid:V5nehGfxQKqylHNOydLsm4lcCaBvlaj3 devn:8:33 vg:vg scan:/dev/sdd1 pvid:ueNNLv8Ifh3mEGxrBIdcmhNBZtMeqiF3 devn:8:49 vg:vg scan:/dev/sde1 pvid:njEZWMeq2a8yhSmHgJc79XeBe4iyzY1l devn:8:65 vg:- scan:/dev/sdf1 pvid:m01NogKn21BQEUWNMivndGAD0re1AneJ devn:8:81 vg:- scan:/dev/sdg1 pvid:rRoTdlEwuBDzxzLt3gSUmYEMoOzhhUhY devn:8:97 vg:- devs_hash: 1094215824 10 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:6633 |
Description of problem: After importing a VG with vgimport on a system with shared storage, the VG is not detected by lvm commands that use the hints file (e.g. vgs, lvs, vgchange, lvchange), until the file is either manually deleted or pvscan --cache is manually run. Version-Release number of selected component (if applicable): Observed on a customer system with lvm2-2.03.14-3.el8.x86_64 Reproduced also with lvm2-2.03.14-6.el8.x86_64 How reproducible: Always Steps to Reproduce: 1. Create 2 VMs sharing a disk via multiple paths. I have been testing with the same disk on the hypervisor presented twice on each VM - with an xml definition containing a <wwn> section and a <shareable/> section, but I expect any kind of shared storage will behave the same way. 2. Setup multipath to handle the paths and create a partition on the mpath device (e.g. mpatha1) (the partition may not be necessary, but I have just been copying the customer environment) 3. Make sure that both VMs see the partition (e.g. restart multipathd on both or reboot the VMs or run kpartx -a /dev/mapper/mpatha) 4. On the 2nd VM: Run vgs so that the hints file gets updated and devs_hash is calculated including all devices. 5. On the 1st VM: create a VG on mpatha1 with some content (e.g. vg01 with a single LV lv01). Then deactivate and export the VG. 6. On the 2nd VM: Run vgimport vg01 to import the VG. Then run vgs to see the newly imported VG. Actual results: vgimport succeeds on the 2nd VM, but the newly imported VG is not visible in vgs (as well as lvs, vgchange and lvchange) Deleting /run/lvm/hints or running pvscan --cache makes the VG visible. Expected results: The customer was expecting that vgimport would make the VG visible and that was my initial expectation too. After reviewing lib/label/hints.c I am not so sure what my expectation should be, as vgimport is listed in the commands that ignore hints. At a first glance it is counter-intuitive that pvscan --cache is necessary along with vgimport (it seems like a redundant step). Additional info: It is not clear to me if there is a real problem or if this is a matter of confusion that can be solved with clearer documentation. Potential outcomes I can think of include: [1] Make it clear (in a KCS?) that after moving PVs to a system it is good to run: pvscan --cache in order to detect new PVs, especially when these are on shared storage [2] Modify vgimport to invalidate the hints file so that any next command recreates it. [3] When trying to reproduce the same steps without multipath (on a shared scsi disk with a partition) I noticed that systemd was automatically running pvscan --cache in the background and the problem didn't reproduce. It is not clear (yet?) to me why an mpath device and a scsi disk behave differently in this case, but maybe it would make sense to have both behave in the same way.