Bug 846376 - 3.1 - VDSM [scalability] beta2 : during connectStoragePool vdsm call 'vgs' twice for each storage domain.
3.1 - VDSM [scalability] beta2 : during connectStoragePool vdsm call 'vgs' tw...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: vdsm (Show other bugs)
6.3
x86_64 Linux
high Severity high
: beta
: ---
Assigned To: Federico Simoncelli
Leonid Natapov
storage scale
: Regression, ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-08-07 11:15 EDT by Omri Hochman
Modified: 2012-12-04 14:04 EST (History)
9 users (show)

See Also:
Fixed In Version: vdsm-4.9.6-32.0
Doc Type: Bug Fix
Doc Text:
Previously, when the VDSM tried to use connectStoragePool it would call 'vgs' twice for each storage domain. This meant activating the host from maintenance mode took an excessively long time. This patch corrects the call, making activating the host much faster with a typical result within less than 30 seconds.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-12-04 14:04:58 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
vdsm.log (551.87 KB, application/octet-stream)
2012-08-07 11:15 EDT, Omri Hochman
no flags Details

  None (edit)
Description Omri Hochman 2012-08-07 11:15:36 EDT
Created attachment 602788 [details]
vdsm.log

VDSM [scalability]: during connectStoragePool vdsm call 'vgs' twice for each storage domain.   

Description: 
*************
Working on iSCSI environment with 30 SD's, connectStoragePool takes more than 3 minutes (3 minutes = Engine timeout) - the reason is that during the connectStoragePool - vdsm calls the 'vgs' command twice for each storage domain,
in total 62 times.  each 'vgs' request takes 4-6 seconds. so by the time the connectStoragePool Return response: True - engine doesn't listen since the 3 minutes timeout on engine side already passed. 

results:
*************
Activate host (from maintenance) takes long time.

Number of time connectStoragePool calls 'vgs' during this tread = 62 :
***********************************************************************
[root@puma07 ~]# xzgrep 'Thread-9381:' /var/log/vdsm/vdsm.log.1.xz | grep '/sbin/lvm vgs' | wc -l
62
  
Error because the connectStoragePool returned after more then 3 minutes:
********************************************************************
Thread-9381::INFO::2012-08-07 15:32:16,199::logUtils::39::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True
Thread-9381::DEBUG::2012-08-07 15:32:16,199::task::1172::TaskManager.Task::(prepare) Task=`bd79a7ff-fb72-4899-afd9-89ee71a50539`::finished: True
Thread-9381::DEBUG::2012-08-07 15:32:16,200::task::588::TaskManager.Task::(_updateState) Task=`bd79a7ff-fb72-4899-afd9-89ee71a50539`::moving from state preparing -> state
 finished
Thread-9381::DEBUG::2012-08-07 15:32:16,200::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-9381::DEBUG::2012-08-07 15:32:16,200::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-9381::DEBUG::2012-08-07 15:32:16,200::task::978::TaskManager.Task::(_decref) Task=`bd79a7ff-fb72-4899-afd9-89ee71a50539`::ref 0 aborting False
Thread-9381::ERROR::2012-08-07 15:32:16,202::SecureXMLRPCServer::73::root::(handle_error) client ('10.35.160.91', 51249)
Traceback (most recent call last):
  File "/usr/lib64/python2.6/SocketServer.py", line 560, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py", line 69, in finish_request
    client_address)
  File "/usr/lib64/python2.6/SocketServer.py", line 322, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib64/python2.6/SocketServer.py", line 617, in __init__
    self.handle()
  File "/usr/lib64/python2.6/BaseHTTPServer.py", line 329, in handle
    self.handle_one_request()
  File "/usr/lib64/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
    method()
  File "/usr/lib64/python2.6/SimpleXMLRPCServer.py", line 498, in do_POST
    self.connection.shutdown(1)
  File "/usr/lib64/python2.6/ssl.py", line 266, in shutdown
    socket.shutdown(self, how)
  File "<string>", line 1, in shutdown
error: [Errno 107] Transport endpoint is not connected


vdsm.log (Shows multiple unnecessary calls to 'vgs' :
*********************************************************
Thread-9381::DEBUG::2012-08-07 15:29:13,899::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%36006016012823000020bae5a42cfe111|36006016012823000060bae5a42cfe111|36006016012823000080bae5a42cfe111|3600601601282300009b5096878cbe111|360060160128230000a0bae5a42cfe111|360060160128230000bb5096878cbe111|360060160128230000c0bae5a42cfe111|360060160128230000db5096878cbe111|360060160128230000e0bae5a42cfe111|360060160128230000fb5096878cbe111|36006016012823000100bae5a42cfe111|3600601601282300011b5096878cbe111|36006016012823000120bae5a42cfe111|360060160128230002265fed572cfe111|360060160128230002465fed572cfe111|3600601601282300024c4e57478cbe111|360060160128230002665fed572cfe111|3600601601282300026c4e57478cbe111|360060160128230002865fed572cfe111|3600601601282300028c4e57478cbe111|360060160128230002a65fed572cfe111|360060160128230002ac4e57478cbe111|360060160128230002c65fed572cfe111|360060160128230002cc4e57478cbe111|360060160128230002e07326742cfe111|360060160128230002e65fed572cfe111|360060160128230002ec4e57478cbe111|360060160128230003065fed572cfe111|3600601601282300030c4e57478cbe111|360060160128230003265fed572cfe111|3600601601282300032c4e57478cbe111|3600601601282300032f94e5b78cbe111|360060160128230003465fed572cfe111|3600601601282300034c4e57478cbe111|3600601601282300034f94e5b78cbe111|360060160128230003665fed572cfe111|3600601601282300036f94e5b78cbe111|360060160128230003865fed572cfe111|3600601601282300038f94e5b78cbe111|360060160128230003af94e5b78cbe111|360060160128230003cf94e5b78cbe111|360060160128230003d4e306142cfe111|360060160128230003ef94e5b78cbe111|360060160128230003f4e306142cfe111|3600601601282300040f94e5b78cbe111|36006016012823000414e306142cfe111|3600601601282300042f94e5b78cbe111|36006016012823000434e306142cfe111|3600601601282300044f94e5b78cbe111|3600601601282300046e16a6178cbe111|3600601601282300046f94e5b78cbe111|3600601601282300048e16a6178cbe111|360060160128230004ae16a6178cbe111|360060160128230004ce16a6178cbe111|360060160128230004ee16a6178cbe111|360060160128230004fc5f0cf72cfe111|3600601601282300050e16a6178cbe111|3600601601282300051c5f0cf72cfe111|3600601601282300052e16a6178cbe111|3600601601282300053c5f0cf72cfe111|3600601601282300055c5f0cf72cfe111|3600601601282300057c5f0cf72cfe111|3600601601282300059c5f0cf72cfe111|360060160128230005bc5f0cf72cfe111|360060160128230005dc5f0cf72cfe111|360060160128230005fc5f0cf72cfe111|3600601601282300061c5f0cf72cfe111|360060160128230006264545578cbe111|3600601601282300063c5f0cf72cfe111|360060160128230006464545578cbe111|3600601601282300065066d6e78cbe111|360060160128230006664545578cbe111|3600601601282300067066d6e78cbe111|360060160128230006864545578cbe111|3600601601282300069066d6e78cbe111|360060160128230006a64545578cbe111|360060160128230006b066d6e78cbe111|360060160128230006c64545578cbe111|360060160128230006d066d6e78cbe111|360060160128230006e64545578cbe111|360060160128230006f066d6e78cbe111|360060160128230007064545578cbe111|3600601601282300071066d6e78cbe111|360060160128230007264545578cbe111|360060160128230007464545578cbe111|360060160128230007664545578cbe111|360060160128230009fda68dc72cfe111|36006016012823000a1da68dc72cfe111|36006016012823000c0f3098442cfe111|36006016012823000c2f3098442cfe111|36006016012823000c4f3098442cfe111|36006016012823000c6f3098442cfe111|36006016012823000c8f3098442cfe111|36006016012823000c9602c468bd5e111|36006016012823000caf3098442cfe111|36006016012823000ccf3098442cfe111|36006016012823000cef3098442cfe111|36006016012823000d0f3098442cfe111|36006016012823000d2f3098442cfe111|36006016012823000d4f3098442cfe111|36006016012823000fe0aae5a42cfe111%\\", \\"r%.*%\\" ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free dcaf3756-1e12-401f-a4b1-8dc943a7f34f' (cwd None)
..
..
..
..
Thread-9381::DEBUG::2012-08-07 15:29:17,701::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%36006016012823000020bae5a42cfe111|36006016012823000060bae5a42cfe111|36006016012823000080bae5a42cfe111|3600601601282300009b5096878cbe111|360060160128230000a0bae5a42cfe111|360060160128230000bb5096878cbe111|360060160128230000c0bae5a42cfe111|360060160128230000db5096878cbe111|360060160128230000e0bae5a42cfe111|360060160128230000fb5096878cbe111|36006016012823000100bae5a42cfe111|3600601601282300011b5096878cbe111|36006016012823000120bae5a42cfe111|360060160128230002265fed572cfe111|360060160128230002465fed572cfe111|3600601601282300024c4e57478cbe111|360060160128230002665fed572cfe111|3600601601282300026c4e57478cbe111|360060160128230002865fed572cfe111|3600601601282300028c4e57478cbe111|360060160128230002a65fed572cfe111|360060160128230002ac4e57478cbe111|360060160128230002c65fed572cfe111|360060160128230002cc4e57478cbe111|360060160128230002e07326742cfe111|360060160128230002e65fed572cfe111|360060160128230002ec4e57478cbe111|360060160128230003065fed572cfe111|3600601601282300030c4e57478cbe111|360060160128230003265fed572cfe111|3600601601282300032c4e57478cbe111|3600601601282300032f94e5b78cbe111|360060160128230003465fed572cfe111|3600601601282300034c4e57478cbe111|3600601601282300034f94e5b78cbe111|360060160128230003665fed572cfe111|3600601601282300036f94e5b78cbe111|360060160128230003865fed572cfe111|3600601601282300038f94e5b78cbe111|360060160128230003af94e5b78cbe111|360060160128230003cf94e5b78cbe111|360060160128230003d4e306142cfe111|360060160128230003ef94e5b78cbe111|360060160128230003f4e306142cfe111|3600601601282300040f94e5b78cbe111|36006016012823000414e306142cfe111|3600601601282300042f94e5b78cbe111|36006016012823000434e306142cfe111|3600601601282300044f94e5b78cbe111|3600601601282300046e16a6178cbe111|3600601601282300046f94e5b78cbe111|3600601601282300048e16a6178cbe111|360060160128230004ae16a6178cbe111|360060160128230004ce16a6178cbe111|360060160128230004ee16a6178cbe111|360060160128230004fc5f0cf72cfe111|3600601601282300050e16a6178cbe111|3600601601282300051c5f0cf72cfe111|3600601601282300052e16a6178cbe111|3600601601282300053c5f0cf72cfe111|3600601601282300055c5f0cf72cfe111|3600601601282300057c5f0cf72cfe111|3600601601282300059c5f0cf72cfe111|360060160128230005bc5f0cf72cfe111|360060160128230005dc5f0cf72cfe111|360060160128230005fc5f0cf72cfe111|3600601601282300061c5f0cf72cfe111|360060160128230006264545578cbe111|3600601601282300063c5f0cf72cfe111|360060160128230006464545578cbe111|3600601601282300065066d6e78cbe111|360060160128230006664545578cbe111|3600601601282300067066d6e78cbe111|360060160128230006864545578cbe111|3600601601282300069066d6e78cbe111|360060160128230006a64545578cbe111|360060160128230006b066d6e78cbe111|360060160128230006c64545578cbe111|360060160128230006d066d6e78cbe111|360060160128230006e64545578cbe111|360060160128230006f066d6e78cbe111|360060160128230007064545578cbe111|3600601601282300071066d6e78cbe111|360060160128230007264545578cbe111|360060160128230007464545578cbe111|360060160128230007664545578cbe111|360060160128230009fda68dc72cfe111|36006016012823000a1da68dc72cfe111|36006016012823000c0f3098442cfe111|36006016012823000c2f3098442cfe111|36006016012823000c4f3098442cfe111|36006016012823000c6f3098442cfe111|36006016012823000c8f3098442cfe111|36006016012823000c9602c468bd5e111|36006016012823000caf3098442cfe111|36006016012823000ccf3098442cfe111|36006016012823000cef3098442cfe111|36006016012823000d0f3098442cfe111|36006016012823000d2f3098442cfe111|36006016012823000d4f3098442cfe111|36006016012823000fe0aae5a42cfe111%\\", \\"r%.*%\\" ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free dcaf3756-1e12-401f-a4b1-8dc943a7f34f' (cwd None)
Thread-9381::DEBUG::2012-08-07 15:29:20,796::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0
Comment 4 Eduardo Warszawski 2012-08-28 09:55:05 EDT
http://gerrit.ovirt.org/#/c/7163/
Comment 5 Ayal Baron 2012-09-02 10:29:03 EDT
Fede, http://gerrit.ovirt.org/#/c/7294 ?
Comment 6 Rami Vaknin 2012-09-03 08:54:12 EDT
Could you please tell how many vgs should be after the above patch? will it be constant or still depends on the number of storage domains?
Comment 8 Ayal Baron 2012-09-30 17:18:24 EDT
(In reply to comment #6)
> Could you please tell how many vgs should be after the above patch? will it
> be constant or still depends on the number of storage domains?

exactly the same number, it's just not in connectStoragePool, it's asynchronously in the domain monitoring threads.

dealing with number of lvm commands will be done in a separate set of patches.
Comment 9 Leonid Natapov 2012-10-14 09:37:53 EDT
vdsm-4.9.6-36.0.el6_3.x86_64. Tested with 30 SD and 3 hosts. ConnectStoragePool was very quick. less than 30 seconds.
Comment 13 errata-xmlrpc 2012-12-04 14:04:58 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-1508.html

Note You need to log in before you can comment on or make changes to this bug.