Bug 1344206
| Summary: | vgck scan all devices even if filter is configured | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Lukas Herbolt <lherbolt> | ||||
| Component: | lvm2 | Assignee: | LVM and device-mapper development team <lvm-team> | ||||
| lvm2 sub component: | Devices, Filtering and Stacking | QA Contact: | cluster-qe <cluster-qe> | ||||
| Status: | CLOSED ERRATA | Docs Contact: | |||||
| Severity: | urgent | ||||||
| Priority: | high | CC: | adevolder, agk, amureini, aperotti, cmarthal, dmoessne, fdeutsch, gklein, heinzm, jbrassow, lherbolt, luvilla, mkalinin, mkarg, msnitzer, nsoffer, pdwyer, prajnoha, prockai, pzhukov, rbednar, rhodain, teigland, zkabelac | ||||
| Version: | 7.2 | Keywords: | ZStream | ||||
| Target Milestone: | rc | Flags: | zkabelac:
needinfo-
|
||||
| Target Release: | --- | ||||||
| Hardware: | All | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | lvm2-2.02.156-1.el7 | Doc Type: | Bug Fix | ||||
| Doc Text: |
Previously, the regular expressions filter was incorrectly positioned within other filters. As a consequence, running certain LVM2 commands, such as vgck and pvs, caused all block devices to be open for scanning, even though the settings in the lvm.conf file specified only devices that were allowed for scanning, while all other devices were supposed to be rejected. With this update, the regular expressions filter is evaluated first, before any other filters that need to open a device. As a result, only the desired devices are opened for scanning in the described situation.
|
Story Points: | --- | ||||
| Clone Of: | |||||||
| : | 1346172 (view as bug list) | Environment: | |||||
| Last Closed: | 2016-11-04 04:21:06 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1346172 | ||||||
| Attachments: |
|
||||||
To be noted here: vgck is not a command to be running periodically in minute range. So performance of this command is not important. However the observed trace might be having similar consequence in other commands like 'vgs'.... When I run an equivalent command using the latest upstream lvm code, only the device accepted by the filter is opened, none others. Perhaps recent improvements in lvm have fixed this problem. I'll try the version you're using to see if that's changed.
(Was including "vgck" as a VG name just a mistake when generating the example trace?
i.e. vgck vgck -vvvv --config 'devices { filter = [ "a|/dev/sda|", "r|.*|" ] }' local-store
Which reports "Cannot process volume group vgck".)
Is lvmetad being used? The trace has: #libdm-config.c:1013 Setting global/use_lvmetad to 1 Using lvm version 2.02.130: If I set use_lvmetad=1, and run vgck with devices/filter, then vgck opens many devices. If I use devices/global_filter, then vgck opens only the accepted device. If I set use_lvmetad=0, then vgck using devices/filter opens many devices, but vgck using devices/global_filter opens only the accepted device. With the latest upstream lvm code: Using use_lvmetad=0 and devices/filter, vgck only opens the accepted device, the same as global_filter. In summary, 2.02.130: use_lvmetad=1, devices/filter: vgck opens many rejected devices use_lvmetad=1, devices/global_filter: vgck opens only accepted device use_lvmetad=0, devices/filter: vgck opens many rejected devices use_lvmetad=0, devices/global_filter: vgck opens only accepted device upstream: use_lvmetad=1, devices/filter: vgck opens only accepted device use_lvmetad=1, devices/global_filter: vgck opens only accepted device use_lvmetad=0, devices/filter: vgck opens only accepted device use_lvmetad=0, devices/global_filter: vgck opens only accepted device I think that opening rejected devices in 2.02.130 is a bug related to checking the device size when we don't need to, but I'm not sure which commit fixed that. Using global_filter instead of filter may be a simple solution. Also, I think you should check if lvmetad is being used, because I don't think RHEV is meant to be using lvmetad. Is there any possible workaround? I've rerun the tests with the version of lvm from rhel 7.2 and get the same results as 2.02.130. There appear to be no patches in the 7.2 version either that would address this issue. > Is there any possible workaround?
As mentioned above, the trivial workaround is to use devices/global_filter instead of devices/filter.
Based on comment 15, I posted a patch for vdsm. Can you test this fixes the issue for the customer? Gil, any chance to do a quick testing for the vdsm patch? A candidate patch that seems to be missing from that particular release is:
commit b77497cbd86a854add3ffa4ce09a35f165c0d2ba
Author: Peter Rajnoha <prajnoha>
Date: Tue Sep 8 15:03:15 2015 +0200
filters: make sure regex filter is evaluated before any filter that needs disk access
(In reply to Alasdair Kergon from comment #18) > A candidate patch that seems to be missing from that particular release is: > > commit b77497cbd86a854add3ffa4ce09a35f165c0d2ba > Author: Peter Rajnoha <prajnoha> > Date: Tue Sep 8 15:03:15 2015 +0200 > > filters: make sure regex filter is evaluated before any filter that > needs disk access Is is possible to get this backported to 7.2.z? What about vdsm usage - regardless of this issue, should we use filter or global_filter in our runtime --config? the patch in comment 18 fixes the problem To test this, create two PVs, each with a different VG, i.e. VG1 on /dev/sdb, VG2 on /dev/sdc. Then, run the following commands without lvmetad: vgck -vvvv |& tee vgck.out1 vgck -vvvv --config 'devices/filter=["a|/dev/sdb|", "r|.*|"]' |& tee vgck.out2 grep Opened vgck.out1 should show both devices are opened grep Opened vgck.out2 should show only /dev/sdb is opened Removing the vdsm patch, as we have a vdsm bug for this. (In reply to Alasdair Kergon from comment #18) > A candidate patch that seems to be missing from that particular release is: > > commit b77497cbd86a854add3ffa4ce09a35f165c0d2ba > Author: Peter Rajnoha <prajnoha> > Date: Tue Sep 8 15:03:15 2015 +0200 > > filters: make sure regex filter is evaluated before any filter that > needs disk access Already included in current 7.3 build - moving to MODIFIED. Verified. vgck now opens only accepted devices when using filter. Tested using strace to avoid lvm2 logging system. As suggested in bug #1346172 ============================================= #strace vgck -vvvv |& tee vgck.out1 #strace vgck -vvvv --config 'devices/filter=["a|/dev/sdb|", "r|.*|"]' |& tee vgck.out2 # grep Open vgck.out1 write(2, "Opened /dev/sda RO O_DIRECT", 27Opened /dev/sda RO O_DIRECT) = 27 write(2, "Opened /dev/sda RO O_DIRECT", 27Opened /dev/sda RO O_DIRECT) = 27 write(2, "Opened /dev/sda RO O_DIRECT", 27Opened /dev/sda RO O_DIRECT) = 27 write(2, "Opened /dev/sda RO O_DIRECT", 27Opened /dev/sda RO O_DIRECT) = 27 write(2, "Opened /dev/vda RO O_DIRECT", 27Opened /dev/vda RO O_DIRECT) = 27 write(2, "Opened /dev/vda RO O_DIRECT", 27Opened /dev/vda RO O_DIRECT) = 27 write(2, "Opened /dev/rhel_virt-252/root R"..., 42Opened /dev/rhel_virt-252/root RO O_DIRECT) = 42 write(2, "Opened /dev/rhel_virt-252/root R"..., 42Opened /dev/rhel_virt-252/root RO O_DIRECT) = 42 write(2, "Opened /dev/rhel_virt-252/root R"..., 42Opened /dev/rhel_virt-252/root RO O_DIRECT) = 42 write(2, "Opened /dev/rhel_virt-252/root R"..., 42Opened /dev/rhel_virt-252/root RO O_DIRECT) = 42 write(2, "Opened /dev/vda1 RO O_DIRECT", 28Opened /dev/vda1 RO O_DIRECT) = 28 write(2, "Opened /dev/vda1 RO O_DIRECT", 28Opened /dev/vda1 RO O_DIRECT) = 28 write(2, "Opened /dev/vda1 RO O_DIRECT", 28Opened /dev/vda1 RO O_DIRECT) = 28 write(2, "Opened /dev/rhel_virt-252/swap R"..., 42Opened /dev/rhel_virt-252/swap RO O_DIRECT) = 42 write(2, "Opened /dev/rhel_virt-252/swap R"..., 42Opened /dev/rhel_virt-252/swap RO O_DIRECT) = 42 write(2, "Opened /dev/rhel_virt-252/swap R"..., 42Opened /dev/rhel_virt-252/swap RO O_DIRECT) = 42 write(2, "Opened /dev/rhel_virt-252/swap R"..., 42Opened /dev/rhel_virt-252/swap RO O_DIRECT) = 42 write(2, "Opened /dev/vda2 RO O_DIRECT", 28Opened /dev/vda2 RO O_DIRECT) = 28 write(2, "Opened /dev/vda2 RO O_DIRECT", 28Opened /dev/vda2 RO O_DIRECT) = 28 write(2, "Opened /dev/vda2 RO O_DIRECT", 28Opened /dev/vda2 RO O_DIRECT) = 28 write(2, "Opened /dev/sdb RO O_DIRECT", 27Opened /dev/sdb RO O_DIRECT) = 27 write(2, "Opened /dev/sdb RO O_DIRECT", 27Opened /dev/sdb RO O_DIRECT) = 27 write(2, "Opened /dev/sdb RO O_DIRECT", 27Opened /dev/sdb RO O_DIRECT) = 27 write(2, "Opened /dev/sdb RO O_DIRECT", 27Opened /dev/sdb RO O_DIRECT) = 27 write(2, "Opened /dev/sdc RO O_DIRECT", 27Opened /dev/sdc RO O_DIRECT) = 27 write(2, "Opened /dev/sdc RO O_DIRECT", 27Opened /dev/sdc RO O_DIRECT) = 27 write(2, "Opened /dev/sdc RO O_DIRECT", 27Opened /dev/sdc RO O_DIRECT) = 27 write(2, "Opened /dev/sdc RO O_DIRECT", 27Opened /dev/sdc RO O_DIRECT) = 27 write(2, "Opened /dev/sdd RO O_DIRECT", 27Opened /dev/sdd RO O_DIRECT) = 27 write(2, "Opened /dev/sdd RO O_DIRECT", 27Opened /dev/sdd RO O_DIRECT) = 27 write(2, "Opened /dev/sdd RO O_DIRECT", 27Opened /dev/sdd RO O_DIRECT) = 27 write(2, "Opened /dev/sdd RO O_DIRECT", 27Opened /dev/sdd RO O_DIRECT) = 27 write(2, "Opened /dev/sde RO O_DIRECT", 27Opened /dev/sde RO O_DIRECT) = 27 write(2, "Opened /dev/sde RO O_DIRECT", 27Opened /dev/sde RO O_DIRECT) = 27 write(2, "Opened /dev/sde RO O_DIRECT", 27Opened /dev/sde RO O_DIRECT) = 27 write(2, "Opened /dev/sde RO O_DIRECT", 27Opened /dev/sde RO O_DIRECT) = 27 write(2, "Opened /dev/sdf RO O_DIRECT", 27Opened /dev/sdf RO O_DIRECT) = 27 write(2, "Opened /dev/sdf RO O_DIRECT", 27Opened /dev/sdf RO O_DIRECT) = 27 write(2, "Opened /dev/sdf RO O_DIRECT", 27Opened /dev/sdf RO O_DIRECT) = 27 write(2, "Opened /dev/sdf RO O_DIRECT", 27Opened /dev/sdf RO O_DIRECT) = 27 write(2, "Opened /dev/sdg RO O_DIRECT", 27Opened /dev/sdg RO O_DIRECT) = 27 write(2, "Opened /dev/sdg RO O_DIRECT", 27Opened /dev/sdg RO O_DIRECT) = 27 write(2, "Opened /dev/sdg RO O_DIRECT", 27Opened /dev/sdg RO O_DIRECT) = 27 write(2, "Opened /dev/sdg RO O_DIRECT", 27Opened /dev/sdg RO O_DIRECT) = 27 write(2, "Opened /dev/sdh RO O_DIRECT", 27Opened /dev/sdh RO O_DIRECT) = 27 write(2, "Opened /dev/sdh RO O_DIRECT", 27Opened /dev/sdh RO O_DIRECT) = 27 write(2, "Opened /dev/sdh RO O_DIRECT", 27Opened /dev/sdh RO O_DIRECT) = 27 write(2, "Opened /dev/sdh RO O_DIRECT", 27Opened /dev/sdh RO O_DIRECT) = 27 write(2, "Opened /dev/sdi RO O_DIRECT", 27Opened /dev/sdi RO O_DIRECT) = 27 write(2, "Opened /dev/sdi RO O_DIRECT", 27Opened /dev/sdi RO O_DIRECT) = 27 write(2, "Opened /dev/sdi RO O_DIRECT", 27Opened /dev/sdi RO O_DIRECT) = 27 write(2, "Opened /dev/sdi RO O_DIRECT", 27Opened /dev/sdi RO O_DIRECT) = 27 write(2, "Opened /dev/sdj RO O_DIRECT", 27Opened /dev/sdj RO O_DIRECT) = 27 write(2, "Opened /dev/sdj RO O_DIRECT", 27Opened /dev/sdj RO O_DIRECT) = 27 write(2, "Opened /dev/sdj RO O_DIRECT", 27Opened /dev/sdj RO O_DIRECT) = 27 write(2, "Opened /dev/sdj RO O_DIRECT", 27Opened /dev/sdj RO O_DIRECT) = 27 write(2, "Opened /dev/sdb RO O_DIRECT", 27Opened /dev/sdb RO O_DIRECT) = 27 write(2, "Opened /dev/vda2 RO O_DIRECT", 28Opened /dev/vda2 RO O_DIRECT) = 28 write(2, "Opened /dev/sdb RO O_DIRECT", 27Opened /dev/sdb RO O_DIRECT) = 27 write(2, "Opened /dev/sda RO O_DIRECT", 27Opened /dev/sda RO O_DIRECT) = 27 write(2, "Opened /dev/vda2 RO O_DIRECT", 28Opened /dev/vda2 RO O_DIRECT) = 28 # grep Open vgck.out2 write(2, "Opened /dev/sdb RO O_DIRECT", 27Opened /dev/sdb RO O_DIRECT) = 27 write(2, "Opened /dev/sdb RO O_DIRECT", 27Opened /dev/sdb RO O_DIRECT) = 27 write(2, "Opened /dev/sdb RO O_DIRECT", 27Opened /dev/sdb RO O_DIRECT) = 27 write(2, "Opened /dev/sdb RO O_DIRECT", 27Opened /dev/sdb RO O_DIRECT) = 27 write(2, "Opened /dev/sdb RO O_DIRECT", 27Opened /dev/sdb RO O_DIRECT) = 27 write(2, "Opened /etc/lvm/lvm.conf RO", 27Opened /etc/lvm/lvm.conf RO) = 27 ============================================= 3.10.0-429.el7.x86_64 lvm2-2.02.156-1.el7 BUILT: Mon Jun 13 10:05:51 CEST 2016 lvm2-libs-2.02.156-1.el7 BUILT: Mon Jun 13 10:05:51 CEST 2016 lvm2-cluster-2.02.156-1.el7 BUILT: Mon Jun 13 10:05:51 CEST 2016 device-mapper-1.02.126-1.el7 BUILT: Mon Jun 13 10:05:51 CEST 2016 device-mapper-libs-1.02.126-1.el7 BUILT: Mon Jun 13 10:05:51 CEST 2016 device-mapper-event-1.02.126-1.el7 BUILT: Mon Jun 13 10:05:51 CEST 2016 device-mapper-event-libs-1.02.126-1.el7 BUILT: Mon Jun 13 10:05:51 CEST 2016 device-mapper-persistent-data-0.6.2-0.1.rc8.el7 BUILT: Wed May 4 09:56:34 CEST 2016 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-1445.html |
Created attachment 1166178 [details] verbose vgck Description of problem: When running vgck with filtered devices the filter is ignored and all devices are opened and scanned for LVM metadata. Version-Release number of selected component (if applicable): lvm2-2.02.130-5.el7.x86_64 How reproducible: Each time you run vgck. Steps to Reproduce: 1. create volume group on device 2. scan for that volume group with: vgck -vvvv --config 'devices { filter = [ "a|/dev/sda|", "r|.*|" ] }' <vgname> 3. You will get result like: ... #device/dev-io.c:533 Opened /dev/sda2 RO O_DIRECT #device/dev-io.c:310 /dev/sda2: size is 975747072 sectors #device/dev-io.c:587 Closed /dev/sda2 #filters/filter-mpath.c:196 /dev/sda2: Device is a partition, using primary device sda for mpath component detection #device/dev-io.c:533 Opened /dev/sda2 RO O_DIRECT #device/dev-io.c:587 Closed /dev/sda2 #device/dev-io.c:310 /dev/sda2: size is 975747072 sectors #device/dev-io.c:533 Opened /dev/sda2 RO O_DIRECT #device/dev-io.c:144 /dev/sda2: block size is 4096 bytes #device/dev-io.c:155 /dev/sda2: physical block size is 512 bytes #device/dev-io.c:587 Closed /dev/sda2 #device/dev-cache.c:1088 Using /dev/sda2 #label/label.c:275 Using cached label for /dev/sda2 #ioctl/libdm-iface.c:1832 dm status (253:2) OF [16384] (*1) sda2 is read correctly as it is part of the filter ... #device/dev-io.c:533 Opened /dev/mapper/mpatha RO O_DIRECT #device/dev-io.c:310 /dev/mapper/mpatha: size is 1468006400 sectors #device/dev-io.c:587 Closed /dev/mapper/mpatha #device/dev-io.c:533 Opened /dev/mapper/mpatha RO O_DIRECT #device/dev-io.c:144 /dev/mapper/mpatha: block size is 4096 bytes #device/dev-io.c:155 /dev/mapper/mpatha: physical block size is 512 bytes #device/dev-io.c:587 Closed /dev/mapper/mpatha #device/dev-io.c:533 Opened /dev/mapper/mpatha RO O_DIRECT #device/dev-io.c:587 Closed /dev/mapper/mpatha #device/dev-io.c:310 /dev/mapper/mpatha: size is 1468006400 sectors #device/dev-io.c:533 Opened /dev/mapper/mpatha RO O_DIRECT #device/dev-io.c:144 /dev/mapper/mpatha: block size is 4096 bytes #device/dev-io.c:155 /dev/mapper/mpatha: physical block size is 512 bytes #device/dev-io.c:587 Closed /dev/mapper/mpatha #filters/filter-regex.c:172 /dev/mapper/mpatha: Skipping (regex) <<< ... #device/dev-io.c:533 Opened /dev/sdf RO O_DIRECT #device/dev-io.c:310 /dev/sdf: size is 104857600 sectors #device/dev-io.c:587 Closed /dev/sdf #ioctl/libdm-iface.c:1832 dm status (253:3) OF [16384] (*1) #filters/filter-mpath.c:253 /dev/sdf: Skipping mpath component device Actual results: So currently all devices are scanned for metadata and skipped after scan, which could lead to significant number of IOs issued to the storage. Expected results: If filter is specified the devices are not opened or read. Additional info: The problem seems to be bound to the filter-regex.c which is called after reading data. ... #device/dev-io.c:587 Closed /dev/mapper/mpatha #filters/filter-regex.c:172 /dev/mapper/mpatha: Skipping (regex) <<< ... Let me know if you need more details about it.