Bug 981055 - Change lvm filter to access RHEV PVs only by full path /dev/mapper/wwid
Change lvm filter to access RHEV PVs only by full path /dev/mapper/wwid
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm (Show other bugs)
3.2.0
Unspecified Unspecified
high Severity high
: ---
: 3.3.0
Assigned To: Yeela Kaplan
Leonid Natapov
storage
:
Depends On:
Blocks: 983599 1019461
  Show dependency treegraph
 
Reported: 2013-07-03 17:54 EDT by Marina
Modified: 2016-02-10 13:33 EST (History)
21 users (show)

See Also:
Fixed In Version: 4.12.0-rc1
Doc Type: Bug Fix
Doc Text:
The LVM filter has been updated to only access physical volumes by full /dev/mapper paths in order to improve performance. This replaces the previous behavior of scanning all devices including logical volumes on physical volumes.
Story Points: ---
Clone Of:
: 983599 (view as bug list)
Environment:
Last Closed: 2014-01-21 11:26:59 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
lvm.py patch (1003 bytes, patch)
2013-07-03 17:56 EDT, Marina
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Bugzilla 990707 None None None Never
Red Hat Knowledge Base (Solution) 450873 None None None Never
oVirt gerrit 16730 None None None Never

  None (edit)
Description Marina 2013-07-03 17:54:42 EDT
Opening a separate bug for this, since the issue described on original BZ#979193 is still not resolved.

Right now the filter applied to lvm command on vdsm is too wide and scans all the devices on the pv mentioned on the filter (specifically the lvs on that pv), while this is not necessary and all RHEV needs is to read LVM metadata, located on the PVs only.

When there are numerous LVs, scanning all of them affect the performance.
See https://bugzilla.redhat.com/show_bug.cgi?id=979193#c12 for details.

As well, when the device is represented with active/passive paths, and the passive paths are disabled, scanning them will result in I/O errors, while there is no reason to scan the passive paths. See https://bugzilla.redhat.com/show_bug.cgi?id=979193#c5 for details.

FYI, the preferred_names parameter used by vdsm on the same command does not affect the filter; it is only applied to the output of the command, after lvm has already scanned all the devices. 

Current filter created by vdsm:
filter = [ 'a\%3600601605ab02c0028c71db4c076e211|3600601605ab02c005a0df4e048a0e211|3600601605ab02c005c0df4e048a0e211|3600601605ab02c005e0df4e048a0e211|3600601605ab02c00600df4e048a0e211|3600601605ab02c00620df4e048a0e211|3600601605ab02c00640df4e048a0e211%\', 'r\%.*%\' ] }

Required filter:
filter = [ \"a|/dev/mapper/3600601605ab02c0028c71db4c076e211|/dev/mapper/3600601605ab02c005a0df4e048a0e211|/dev/mapper/3600601605ab02c005c0df4e048a0e211|/dev/mapper/3600601605ab02c005e0df4e048a0e211|/dev/mapper/3600601605ab02c00600df4e048a0e211|/dev/mapper/3600601605ab02c00620df4e048a0e211|/dev/mapper/3600601605ab02c00640df4e048a0e211|\", \"r|.*|\" ] }

The patch is attached.
Comment 1 Marina 2013-07-03 17:56:53 EDT
Created attachment 768496 [details]
lvm.py patch
Comment 2 Marina 2013-07-03 17:59:54 EDT
(In reply to Yeela Kaplan from comment #19)

> Regarding the patch, it's enough to make only the following change: 
> <             devs.append('/dev/mapper/' + strippedDev.replace(r'\x',
> r'\\x'))
> ---
> >             devs.append(strippedDev.replace(r'\x', r'\\x'))
> 
> and not all three lines.

Yaela, I was not sure about it.
What do the symbols '\' and '%' mean? I could not find them on lvm man page.
Doing it in the way I did assures we are taking only the devices on the list.
Can you please explain the difference?
Comment 3 Marina 2013-07-05 16:22:49 EDT
FYI, from customer's testing, lvm operations and waiting on mutex time were significantly reduced after applying the filter.
Here is an example of createVolume task running after using the new lvm filter.
Not the waiting time ->  7 sec now compared to almost a minute before, as on:
https://bugzilla.redhat.com/show_bug.cgi?id=979193#c0

And with the new filter:
$ grep 24559e83-4f6e-4516-9e58-3ffc5db0e21f vdsm.log | grep OperationMutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:32,741::lvm::414::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:43,776::lvm::414::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:50,219::lvm::447::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:53,138::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>> less then a sec
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:53,575::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:53,590::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:53,711::lvm::414::OperationMutex::(_reloadlvs) Got the operational mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:01,326::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>> 3 sec
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:04,377::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:04,380::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:04,388::lvm::414::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:16,036::lvm::447::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:17,358::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>> less then a sec
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:17,733::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:17,763::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:17,780::lvm::414::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:21,141::lvm::447::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:21,144::lvm::414::OperationMutex::(_reloadlvs) Operation 'lvm invalidate operation' is holding the operation mutex, waiting...
>> less then a sec
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:21,145::lvm::414::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:37,830::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>> 1.5 sec
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:39,510::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:40,375::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>> 3 sec
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:43,940::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex

>> total: about 7 seconds
Comment 7 Leonid Natapov 2013-07-30 09:14:35 EDT
vdsm-4.12.0-rc3.12.git139ec2f.el6ev.x86_64 .
Filter points  to /dev/mapper full path
Comment 8 thunt 2013-08-07 14:27:38 EDT
We implemented this in 3.1 with the following filter which ignores all of the RHEV LV's

 filter = [ "a|/dev/[hsv]d.*|", "r|/dev/mapper/.*-.*|", "a|/dev/mapper/.*|", "r|.*|" ]

Note the inclusion of local drives.
Comment 9 Yaniv Kaul 2013-09-23 01:38:16 EDT
I reckon it means you do not support EMC PowerPath (/dev/emcpower*) ?
Comment 10 Yaniv Kaul 2013-09-23 02:10:20 EDT
(In reply to Yaniv Kaul from comment #9)
> I reckon it means you do not support EMC PowerPath (/dev/emcpower*) ?

Disregard - vdsm relies too much on multipath for its work. Another path dependency won't matter. :(
Comment 11 Charlie 2013-11-27 19:26:08 EST
This bug is currently attached to errata RHBA-2013:15291. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to 
minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.
* Consequence: What happens when the bug presents.
* Fix: What was done to fix the bug.
* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes 

Thanks in advance.
Comment 12 errata-xmlrpc 2014-01-21 11:26:59 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0040.html

Note You need to log in before you can comment on or make changes to this bug.