Opening a separate bug for this, since the issue described on original BZ#979193 is still not resolved. Right now the filter applied to lvm command on vdsm is too wide and scans all the devices on the pv mentioned on the filter (specifically the lvs on that pv), while this is not necessary and all RHEV needs is to read LVM metadata, located on the PVs only. When there are numerous LVs, scanning all of them affect the performance. See https://bugzilla.redhat.com/show_bug.cgi?id=979193#c12 for details. As well, when the device is represented with active/passive paths, and the passive paths are disabled, scanning them will result in I/O errors, while there is no reason to scan the passive paths. See https://bugzilla.redhat.com/show_bug.cgi?id=979193#c5 for details. FYI, the preferred_names parameter used by vdsm on the same command does not affect the filter; it is only applied to the output of the command, after lvm has already scanned all the devices. Current filter created by vdsm: filter = [ 'a\%3600601605ab02c0028c71db4c076e211|3600601605ab02c005a0df4e048a0e211|3600601605ab02c005c0df4e048a0e211|3600601605ab02c005e0df4e048a0e211|3600601605ab02c00600df4e048a0e211|3600601605ab02c00620df4e048a0e211|3600601605ab02c00640df4e048a0e211%\', 'r\%.*%\' ] } Required filter: filter = [ \"a|/dev/mapper/3600601605ab02c0028c71db4c076e211|/dev/mapper/3600601605ab02c005a0df4e048a0e211|/dev/mapper/3600601605ab02c005c0df4e048a0e211|/dev/mapper/3600601605ab02c005e0df4e048a0e211|/dev/mapper/3600601605ab02c00600df4e048a0e211|/dev/mapper/3600601605ab02c00620df4e048a0e211|/dev/mapper/3600601605ab02c00640df4e048a0e211|\", \"r|.*|\" ] } The patch is attached.
Created attachment 768496 [details] lvm.py patch
(In reply to Yeela Kaplan from comment #19) > Regarding the patch, it's enough to make only the following change: > < devs.append('/dev/mapper/' + strippedDev.replace(r'\x', > r'\\x')) > --- > > devs.append(strippedDev.replace(r'\x', r'\\x')) > > and not all three lines. Yaela, I was not sure about it. What do the symbols '\' and '%' mean? I could not find them on lvm man page. Doing it in the way I did assures we are taking only the devices on the list. Can you please explain the difference?
FYI, from customer's testing, lvm operations and waiting on mutex time were significantly reduced after applying the filter. Here is an example of createVolume task running after using the new lvm filter. Not the waiting time -> 7 sec now compared to almost a minute before, as on: https://bugzilla.redhat.com/show_bug.cgi?id=979193#c0 And with the new filter: $ grep 24559e83-4f6e-4516-9e58-3ffc5db0e21f vdsm.log | grep OperationMutex 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:32,741::lvm::414::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:43,776::lvm::414::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:50,219::lvm::447::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:53,138::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting... >> less then a sec 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:53,575::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:53,590::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:39:53,711::lvm::414::OperationMutex::(_reloadlvs) Got the operational mutex 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:01,326::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm reload operation' is holding the operation mutex, waiting... >> 3 sec 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:04,377::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:04,380::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:04,388::lvm::414::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:16,036::lvm::447::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:17,358::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm reload operation' is holding the operation mutex, waiting... >> less then a sec 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:17,733::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:17,763::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:17,780::lvm::414::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:21,141::lvm::447::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:21,144::lvm::414::OperationMutex::(_reloadlvs) Operation 'lvm invalidate operation' is holding the operation mutex, waiting... >> less then a sec 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:21,145::lvm::414::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:37,830::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm reload operation' is holding the operation mutex, waiting... >> 1.5 sec 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:39,510::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:40,375::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm reload operation' is holding the operation mutex, waiting... >> 3 sec 24559e83-4f6e-4516-9e58-3ffc5db0e21f::DEBUG::2013-07-04 00:40:43,940::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >> total: about 7 seconds
vdsm-4.12.0-rc3.12.git139ec2f.el6ev.x86_64 . Filter points to /dev/mapper full path
We implemented this in 3.1 with the following filter which ignores all of the RHEV LV's filter = [ "a|/dev/[hsv]d.*|", "r|/dev/mapper/.*-.*|", "a|/dev/mapper/.*|", "r|.*|" ] Note the inclusion of local drives.
I reckon it means you do not support EMC PowerPath (/dev/emcpower*) ?
(In reply to Yaniv Kaul from comment #9) > I reckon it means you do not support EMC PowerPath (/dev/emcpower*) ? Disregard - vdsm relies too much on multipath for its work. Another path dependency won't matter. :(
This bug is currently attached to errata RHBA-2013:15291. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag. Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information: * Cause: What actions or circumstances cause this bug to present. * Consequence: What happens when the bug presents. * Fix: What was done to fix the bug. * Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore') Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug. For further details on the Cause, Consequence, Fix, Result format please refer to: https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes Thanks in advance.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-0040.html