Previously, when the VDSM tried to use connectStoragePool it would call 'vgs' twice for each storage domain. This meant activating the host from maintenance mode took an excessively long time. This patch corrects the call, making activating the host much faster with a typical result within less than 30 seconds.
(In reply to comment #6)
> Could you please tell how many vgs should be after the above patch? will it
> be constant or still depends on the number of storage domains?
exactly the same number, it's just not in connectStoragePool, it's asynchronously in the domain monitoring threads.
dealing with number of lvm commands will be done in a separate set of patches.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
http://rhn.redhat.com/errata/RHSA-2012-1508.html
Created attachment 602788 [details] vdsm.log VDSM [scalability]: during connectStoragePool vdsm call 'vgs' twice for each storage domain. Description: ************* Working on iSCSI environment with 30 SD's, connectStoragePool takes more than 3 minutes (3 minutes = Engine timeout) - the reason is that during the connectStoragePool - vdsm calls the 'vgs' command twice for each storage domain, in total 62 times. each 'vgs' request takes 4-6 seconds. so by the time the connectStoragePool Return response: True - engine doesn't listen since the 3 minutes timeout on engine side already passed. results: ************* Activate host (from maintenance) takes long time. Number of time connectStoragePool calls 'vgs' during this tread = 62 : *********************************************************************** [root@puma07 ~]# xzgrep 'Thread-9381:' /var/log/vdsm/vdsm.log.1.xz | grep '/sbin/lvm vgs' | wc -l 62 Error because the connectStoragePool returned after more then 3 minutes: ******************************************************************** Thread-9381::INFO::2012-08-07 15:32:16,199::logUtils::39::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True Thread-9381::DEBUG::2012-08-07 15:32:16,199::task::1172::TaskManager.Task::(prepare) Task=`bd79a7ff-fb72-4899-afd9-89ee71a50539`::finished: True Thread-9381::DEBUG::2012-08-07 15:32:16,200::task::588::TaskManager.Task::(_updateState) Task=`bd79a7ff-fb72-4899-afd9-89ee71a50539`::moving from state preparing -> state finished Thread-9381::DEBUG::2012-08-07 15:32:16,200::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-9381::DEBUG::2012-08-07 15:32:16,200::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-9381::DEBUG::2012-08-07 15:32:16,200::task::978::TaskManager.Task::(_decref) Task=`bd79a7ff-fb72-4899-afd9-89ee71a50539`::ref 0 aborting False Thread-9381::ERROR::2012-08-07 15:32:16,202::SecureXMLRPCServer::73::root::(handle_error) client ('10.35.160.91', 51249) Traceback (most recent call last): File "/usr/lib64/python2.6/SocketServer.py", line 560, in process_request_thread self.finish_request(request, client_address) File "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py", line 69, in finish_request client_address) File "/usr/lib64/python2.6/SocketServer.py", line 322, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib64/python2.6/SocketServer.py", line 617, in __init__ self.handle() File "/usr/lib64/python2.6/BaseHTTPServer.py", line 329, in handle self.handle_one_request() File "/usr/lib64/python2.6/BaseHTTPServer.py", line 323, in handle_one_request method() File "/usr/lib64/python2.6/SimpleXMLRPCServer.py", line 498, in do_POST self.connection.shutdown(1) File "/usr/lib64/python2.6/ssl.py", line 266, in shutdown socket.shutdown(self, how) File "<string>", line 1, in shutdown error: [Errno 107] Transport endpoint is not connected vdsm.log (Shows multiple unnecessary calls to 'vgs' : ********************************************************* Thread-9381::DEBUG::2012-08-07 15:29:13,899::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%36006016012823000020bae5a42cfe111|36006016012823000060bae5a42cfe111|36006016012823000080bae5a42cfe111|3600601601282300009b5096878cbe111|360060160128230000a0bae5a42cfe111|360060160128230000bb5096878cbe111|360060160128230000c0bae5a42cfe111|360060160128230000db5096878cbe111|360060160128230000e0bae5a42cfe111|360060160128230000fb5096878cbe111|36006016012823000100bae5a42cfe111|3600601601282300011b5096878cbe111|36006016012823000120bae5a42cfe111|360060160128230002265fed572cfe111|360060160128230002465fed572cfe111|3600601601282300024c4e57478cbe111|360060160128230002665fed572cfe111|3600601601282300026c4e57478cbe111|360060160128230002865fed572cfe111|3600601601282300028c4e57478cbe111|360060160128230002a65fed572cfe111|360060160128230002ac4e57478cbe111|360060160128230002c65fed572cfe111|360060160128230002cc4e57478cbe111|360060160128230002e07326742cfe111|360060160128230002e65fed572cfe111|360060160128230002ec4e57478cbe111|360060160128230003065fed572cfe111|3600601601282300030c4e57478cbe111|360060160128230003265fed572cfe111|3600601601282300032c4e57478cbe111|3600601601282300032f94e5b78cbe111|360060160128230003465fed572cfe111|3600601601282300034c4e57478cbe111|3600601601282300034f94e5b78cbe111|360060160128230003665fed572cfe111|3600601601282300036f94e5b78cbe111|360060160128230003865fed572cfe111|3600601601282300038f94e5b78cbe111|360060160128230003af94e5b78cbe111|360060160128230003cf94e5b78cbe111|360060160128230003d4e306142cfe111|360060160128230003ef94e5b78cbe111|360060160128230003f4e306142cfe111|3600601601282300040f94e5b78cbe111|36006016012823000414e306142cfe111|3600601601282300042f94e5b78cbe111|36006016012823000434e306142cfe111|3600601601282300044f94e5b78cbe111|3600601601282300046e16a6178cbe111|3600601601282300046f94e5b78cbe111|3600601601282300048e16a6178cbe111|360060160128230004ae16a6178cbe111|360060160128230004ce16a6178cbe111|360060160128230004ee16a6178cbe111|360060160128230004fc5f0cf72cfe111|3600601601282300050e16a6178cbe111|3600601601282300051c5f0cf72cfe111|3600601601282300052e16a6178cbe111|3600601601282300053c5f0cf72cfe111|3600601601282300055c5f0cf72cfe111|3600601601282300057c5f0cf72cfe111|3600601601282300059c5f0cf72cfe111|360060160128230005bc5f0cf72cfe111|360060160128230005dc5f0cf72cfe111|360060160128230005fc5f0cf72cfe111|3600601601282300061c5f0cf72cfe111|360060160128230006264545578cbe111|3600601601282300063c5f0cf72cfe111|360060160128230006464545578cbe111|3600601601282300065066d6e78cbe111|360060160128230006664545578cbe111|3600601601282300067066d6e78cbe111|360060160128230006864545578cbe111|3600601601282300069066d6e78cbe111|360060160128230006a64545578cbe111|360060160128230006b066d6e78cbe111|360060160128230006c64545578cbe111|360060160128230006d066d6e78cbe111|360060160128230006e64545578cbe111|360060160128230006f066d6e78cbe111|360060160128230007064545578cbe111|3600601601282300071066d6e78cbe111|360060160128230007264545578cbe111|360060160128230007464545578cbe111|360060160128230007664545578cbe111|360060160128230009fda68dc72cfe111|36006016012823000a1da68dc72cfe111|36006016012823000c0f3098442cfe111|36006016012823000c2f3098442cfe111|36006016012823000c4f3098442cfe111|36006016012823000c6f3098442cfe111|36006016012823000c8f3098442cfe111|36006016012823000c9602c468bd5e111|36006016012823000caf3098442cfe111|36006016012823000ccf3098442cfe111|36006016012823000cef3098442cfe111|36006016012823000d0f3098442cfe111|36006016012823000d2f3098442cfe111|36006016012823000d4f3098442cfe111|36006016012823000fe0aae5a42cfe111%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free dcaf3756-1e12-401f-a4b1-8dc943a7f34f' (cwd None) .. .. .. .. Thread-9381::DEBUG::2012-08-07 15:29:17,701::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%36006016012823000020bae5a42cfe111|36006016012823000060bae5a42cfe111|36006016012823000080bae5a42cfe111|3600601601282300009b5096878cbe111|360060160128230000a0bae5a42cfe111|360060160128230000bb5096878cbe111|360060160128230000c0bae5a42cfe111|360060160128230000db5096878cbe111|360060160128230000e0bae5a42cfe111|360060160128230000fb5096878cbe111|36006016012823000100bae5a42cfe111|3600601601282300011b5096878cbe111|36006016012823000120bae5a42cfe111|360060160128230002265fed572cfe111|360060160128230002465fed572cfe111|3600601601282300024c4e57478cbe111|360060160128230002665fed572cfe111|3600601601282300026c4e57478cbe111|360060160128230002865fed572cfe111|3600601601282300028c4e57478cbe111|360060160128230002a65fed572cfe111|360060160128230002ac4e57478cbe111|360060160128230002c65fed572cfe111|360060160128230002cc4e57478cbe111|360060160128230002e07326742cfe111|360060160128230002e65fed572cfe111|360060160128230002ec4e57478cbe111|360060160128230003065fed572cfe111|3600601601282300030c4e57478cbe111|360060160128230003265fed572cfe111|3600601601282300032c4e57478cbe111|3600601601282300032f94e5b78cbe111|360060160128230003465fed572cfe111|3600601601282300034c4e57478cbe111|3600601601282300034f94e5b78cbe111|360060160128230003665fed572cfe111|3600601601282300036f94e5b78cbe111|360060160128230003865fed572cfe111|3600601601282300038f94e5b78cbe111|360060160128230003af94e5b78cbe111|360060160128230003cf94e5b78cbe111|360060160128230003d4e306142cfe111|360060160128230003ef94e5b78cbe111|360060160128230003f4e306142cfe111|3600601601282300040f94e5b78cbe111|36006016012823000414e306142cfe111|3600601601282300042f94e5b78cbe111|36006016012823000434e306142cfe111|3600601601282300044f94e5b78cbe111|3600601601282300046e16a6178cbe111|3600601601282300046f94e5b78cbe111|3600601601282300048e16a6178cbe111|360060160128230004ae16a6178cbe111|360060160128230004ce16a6178cbe111|360060160128230004ee16a6178cbe111|360060160128230004fc5f0cf72cfe111|3600601601282300050e16a6178cbe111|3600601601282300051c5f0cf72cfe111|3600601601282300052e16a6178cbe111|3600601601282300053c5f0cf72cfe111|3600601601282300055c5f0cf72cfe111|3600601601282300057c5f0cf72cfe111|3600601601282300059c5f0cf72cfe111|360060160128230005bc5f0cf72cfe111|360060160128230005dc5f0cf72cfe111|360060160128230005fc5f0cf72cfe111|3600601601282300061c5f0cf72cfe111|360060160128230006264545578cbe111|3600601601282300063c5f0cf72cfe111|360060160128230006464545578cbe111|3600601601282300065066d6e78cbe111|360060160128230006664545578cbe111|3600601601282300067066d6e78cbe111|360060160128230006864545578cbe111|3600601601282300069066d6e78cbe111|360060160128230006a64545578cbe111|360060160128230006b066d6e78cbe111|360060160128230006c64545578cbe111|360060160128230006d066d6e78cbe111|360060160128230006e64545578cbe111|360060160128230006f066d6e78cbe111|360060160128230007064545578cbe111|3600601601282300071066d6e78cbe111|360060160128230007264545578cbe111|360060160128230007464545578cbe111|360060160128230007664545578cbe111|360060160128230009fda68dc72cfe111|36006016012823000a1da68dc72cfe111|36006016012823000c0f3098442cfe111|36006016012823000c2f3098442cfe111|36006016012823000c4f3098442cfe111|36006016012823000c6f3098442cfe111|36006016012823000c8f3098442cfe111|36006016012823000c9602c468bd5e111|36006016012823000caf3098442cfe111|36006016012823000ccf3098442cfe111|36006016012823000cef3098442cfe111|36006016012823000d0f3098442cfe111|36006016012823000d2f3098442cfe111|36006016012823000d4f3098442cfe111|36006016012823000fe0aae5a42cfe111%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free dcaf3756-1e12-401f-a4b1-8dc943a7f34f' (cwd None) Thread-9381::DEBUG::2012-08-07 15:29:20,796::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0