Bug 981055

Summary: Change lvm filter to access RHEV PVs only by full path /dev/mapper/wwid
Product: Red Hat Enterprise Virtualization Manager Reporter: Marina Kalinin <mkalinin>
Component: vdsmAssignee: Yeela Kaplan <ykaplan>
Status: CLOSED ERRATA QA Contact: Leonid Natapov <lnatapov>
Severity: high Docs Contact:
Priority: high    
Version: 3.2.0CC: abaron, amureini, bazulay, byount, cpelland, iheim, jcoscia, jkt, jwest, lpeer, lyarwood, mykaul, oourfali, pzhukov, rick.beldin, scohen, sputhenp, tcarlin, thunt, yeylon, ykaplan
Target Milestone: ---   
Target Release: 3.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: storage
Fixed In Version: 4.12.0-rc1 Doc Type: Bug Fix
Doc Text:
The LVM filter has been updated to only access physical volumes by full /dev/mapper paths in order to improve performance. This replaces the previous behavior of scanning all devices including logical volumes on physical volumes.
 Story Points: ---
Clone Of:
: 983599 (view as bug list) Environment:
Last Closed: 2014-01-21 16:26:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 983599, 1019461    
Attachments:
Description Flags
lvm.py patch none

Description Marina Kalinin 2013-07-03 21:54:42 UTC 
Opening a separate bug for this, since the issue described on original BZ#979193 is still not resolved.

Right now the filter applied to lvm command on vdsm is too wide and scans all the devices on the pv mentioned on the filter (specifically the lvs on that pv), while this is not necessary and all RHEV needs is to read LVM metadata, located on the PVs only.

When there are numerous LVs, scanning all of them affect the performance.
See https://bugzilla.redhat.com/show_bug.cgi?id=979193#c12 for details.

As well, when the device is represented with active/passive paths, and the passive paths are disabled, scanning them will result in I/O errors, while there is no reason to scan the passive paths. See https://bugzilla.redhat.com/show_bug.cgi?id=979193#c5 for details.

FYI, the preferred_names parameter used by vdsm on the same command does not affect the filter; it is only applied to the output of the command, after lvm has already scanned all the devices. 

Current filter created by vdsm:
filter = [ 'a\%3600601605ab02c0028c71db4c076e211|3600601605ab02c005a0df4e048a0e211|3600601605ab02c005c0df4e048a0e211|3600601605ab02c005e0df4e048a0e211|3600601605ab02c00600df4e048a0e211|3600601605ab02c00620df4e048a0e211|3600601605ab02c00640df4e048a0e211%\', 'r\%.*%\' ] }

Required filter:
filter = [ \"a|/dev/mapper/3600601605ab02c0028c71db4c076e211|/dev/mapper/3600601605ab02c005a0df4e048a0e211|/dev/mapper/3600601605ab02c005c0df4e048a0e211|/dev/mapper/3600601605ab02c005e0df4e048a0e211|/dev/mapper/3600601605ab02c00600df4e048a0e211|/dev/mapper/3600601605ab02c00620df4e048a0e211|/dev/mapper/3600601605ab02c00640df4e048a0e211|\", \"r|.*|\" ] }

The patch is attached.
Comment 1 Marina Kalinin 2013-07-03 21:56:53 UTC 
Created attachment 768496 [details]
lvm.py patch
Comment 2 Marina Kalinin 2013-07-03 21:59:54 UTC 
(In reply to Yeela Kaplan from comment #19)

> Regarding the patch, it's enough to make only the following change: 
> <             devs.append('/dev/mapper/' + strippedDev.replace(r'\x',
> r'\\x'))
> ---
> >             devs.append(strippedDev.replace(r'\x', r'\\x'))
> 
> and not all three lines.

Yaela, I was not sure about it.
What do the symbols '\' and '%' mean? I could not find them on lvm man page.
Doing it in the way I did assures we are taking only the devices on the list.
Can you please explain the difference?
Comment 3 Marina Kalinin 2013-07-05 20:22:49 UTC 
FYI, from customer's testing, lvm operations and waiting on mutex time were significantly reduced after applying the filter.
Here is an example of createVolume task running after using the new lvm filter.
Not the waiting time ->  7 sec now compared to almost a minute before, as on:
https://bugzilla.redhat.com/show_bug.cgi?id=979193#c0

And with the new filter:
$ grep 24559e83-4f6e-4516-9e58-3ffc5db0e21f vdsm.log | grep OperationMutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:32,741::lvm::414::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:43,776::lvm::414::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:50,219::lvm::447::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:53,138::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>> less then a sec
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:53,575::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:53,590::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:53,711::lvm::414::OperationMutex::(_reloadlvs) Got the operational mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:01,326::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>> 3 sec
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:04,377::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:04,380::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:04,388::lvm::414::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:16,036::lvm::447::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:17,358::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>> less then a sec
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:17,733::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:17,763::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:17,780::lvm::414::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:21,141::lvm::447::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:21,144::lvm::414::OperationMutex::(_reloadlvs) Operation 'lvm invalidate operation' is holding the operation mutex, waiting...
>> less then a sec
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:21,145::lvm::414::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:37,830::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>> 1.5 sec
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:39,510::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:40,375::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>> 3 sec
24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:43,940::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex

>> total: about 7 seconds
Comment 7 Leonid Natapov 2013-07-30 13:14:35 UTC 
vdsm-4.12.0-rc3.12.git139ec2f.el6ev.x86_64 .
Filter points  to /dev/mapper full path
Comment 8 thunt 2013-08-07 18:27:38 UTC 
We implemented this in 3.1 with the following filter which ignores all of the RHEV LV's

 filter = [ "a|/dev/[hsv]d.*|", "r|/dev/mapper/.*-.*|", "a|/dev/mapper/.*|", "r|.*|" ]

Note the inclusion of local drives.
Comment 9 Yaniv Kaul 2013-09-23 05:38:16 UTC 
I reckon it means you do not support EMC PowerPath (/dev/emcpower*) ?
Comment 10 Yaniv Kaul 2013-09-23 06:10:20 UTC 
(In reply to Yaniv Kaul from comment #9)
> I reckon it means you do not support EMC PowerPath (/dev/emcpower*) ?

Disregard - vdsm relies too much on multipath for its work. Another path dependency won't matter. :(
Comment 11 Charlie 2013-11-28 00:26:08 UTC 
This bug is currently attached to errata RHBA-2013:15291. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to 
minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.
* Consequence: What happens when the bug presents.
* Fix: What was done to fix the bug.
* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes 

Thanks in advance.
Comment 12 errata-xmlrpc 2014-01-21 16:26:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0040.html