RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 846376 - 3.1 - VDSM [scalability] beta2 : during connectStoragePool vdsm call 'vgs' twice for each storage domain.
Summary: 3.1 - VDSM [scalability] beta2 : during connectStoragePool vdsm call 'vgs' tw...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: vdsm
Version: 6.3
Hardware: x86_64
OS: Linux
high
high
Target Milestone: beta
: ---
Assignee: Federico Simoncelli
QA Contact: Leonid Natapov
URL:
Whiteboard: storage scale
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-08-07 15:15 UTC by Omri Hochman
Modified: 2022-07-09 05:37 UTC (History)
9 users (show)

Fixed In Version: vdsm-4.9.6-32.0
Doc Type: Bug Fix
Doc Text:
Previously, when the VDSM tried to use connectStoragePool it would call 'vgs' twice for each storage domain. This meant activating the host from maintenance mode took an excessively long time. This patch corrects the call, making activating the host much faster with a typical result within less than 30 seconds.
Clone Of:
Environment:
Last Closed: 2012-12-04 19:04:58 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
vdsm.log (551.87 KB, application/octet-stream)
2012-08-07 15:15 UTC, Omri Hochman 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2012:1508 0 normal SHIPPED_LIVE Important: rhev-3.1.0 vdsm security, bug fix, and enhancement update 2012-12-04 23:48:05 UTC

Description Omri Hochman 2012-08-07 15:15:36 UTC 
Created attachment 602788 [details]
vdsm.log

VDSM [scalability]: during connectStoragePool vdsm call 'vgs' twice for each storage domain.   

Description: 
*************
Working on iSCSI environment with 30 SD's, connectStoragePool takes more than 3 minutes (3 minutes = Engine timeout) - the reason is that during the connectStoragePool - vdsm calls the 'vgs' command twice for each storage domain,
in total 62 times.  each 'vgs' request takes 4-6 seconds. so by the time the connectStoragePool Return response: True - engine doesn't listen since the 3 minutes timeout on engine side already passed. 

results:
*************
Activate host (from maintenance) takes long time.

Number of time connectStoragePool calls 'vgs' during this tread = 62 :
***********************************************************************
[root@puma07 ~]# xzgrep 'Thread-9381:' /var/log/vdsm/vdsm.log.1.xz | grep '/sbin/lvm vgs' | wc -l
62
  
Error because the connectStoragePool returned after more then 3 minutes:
********************************************************************
Thread-9381::INFO::2012-08-07 15:32:16,199::logUtils::39::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True
Thread-9381::DEBUG::2012-08-07 15:32:16,199::task::1172::TaskManager.Task::(prepare) Task=`bd79a7ff-fb72-4899-afd9-89ee71a50539`::finished: True
Thread-9381::DEBUG::2012-08-07 15:32:16,200::task::588::TaskManager.Task::(_updateState) Task=`bd79a7ff-fb72-4899-afd9-89ee71a50539`::moving from state preparing -> state
 finished
Thread-9381::DEBUG::2012-08-07 15:32:16,200::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-9381::DEBUG::2012-08-07 15:32:16,200::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-9381::DEBUG::2012-08-07 15:32:16,200::task::978::TaskManager.Task::(_decref) Task=`bd79a7ff-fb72-4899-afd9-89ee71a50539`::ref 0 aborting False
Thread-9381::ERROR::2012-08-07 15:32:16,202::SecureXMLRPCServer::73::root::(handle_error) client ('10.35.160.91', 51249)
Traceback (most recent call last):
  File "/usr/lib64/python2.6/SocketServer.py", line 560, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py", line 69, in finish_request
    client_address)
  File "/usr/lib64/python2.6/SocketServer.py", line 322, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib64/python2.6/SocketServer.py", line 617, in __init__
    self.handle()
  File "/usr/lib64/python2.6/BaseHTTPServer.py", line 329, in handle
    self.handle_one_request()
  File "/usr/lib64/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
    method()
  File "/usr/lib64/python2.6/SimpleXMLRPCServer.py", line 498, in do_POST
    self.connection.shutdown(1)
  File "/usr/lib64/python2.6/ssl.py", line 266, in shutdown
    socket.shutdown(self, how)
  File "<string>", line 1, in shutdown
error: [Errno 107] Transport endpoint is not connected


vdsm.log (Shows multiple unnecessary calls to 'vgs' :
*********************************************************
Thread-9381::DEBUG::2012-08-07 15:29:13,899::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%36006016012823000020bae5a42cfe111|36006016012823000060bae5a42cfe111|36006016012823000080bae5a42cfe111|3600601601282300009b5096878cbe111|360060160128230000a0bae5a42cfe111|360060160128230000bb5096878cbe111|360060160128230000c0bae5a42cfe111|360060160128230000db5096878cbe111|360060160128230000e0bae5a42cfe111|360060160128230000fb5096878cbe111|36006016012823000100bae5a42cfe111|3600601601282300011b5096878cbe111|36006016012823000120bae5a42cfe111|360060160128230002265fed572cfe111|360060160128230002465fed572cfe111|3600601601282300024c4e57478cbe111|360060160128230002665fed572cfe111|3600601601282300026c4e57478cbe111|360060160128230002865fed572cfe111|3600601601282300028c4e57478cbe111|360060160128230002a65fed572cfe111|360060160128230002ac4e57478cbe111|360060160128230002c65fed572cfe111|360060160128230002cc4e57478cbe111|360060160128230002e07326742cfe111|360060160128230002e65fed572cfe111|360060160128230002ec4e57478cbe111|360060160128230003065fed572cfe111|3600601601282300030c4e57478cbe111|360060160128230003265fed572cfe111|3600601601282300032c4e57478cbe111|3600601601282300032f94e5b78cbe111|360060160128230003465fed572cfe111|3600601601282300034c4e57478cbe111|3600601601282300034f94e5b78cbe111|360060160128230003665fed572cfe111|3600601601282300036f94e5b78cbe111|360060160128230003865fed572cfe111|3600601601282300038f94e5b78cbe111|360060160128230003af94e5b78cbe111|360060160128230003cf94e5b78cbe111|360060160128230003d4e306142cfe111|360060160128230003ef94e5b78cbe111|360060160128230003f4e306142cfe111|3600601601282300040f94e5b78cbe111|36006016012823000414e306142cfe111|3600601601282300042f94e5b78cbe111|36006016012823000434e306142cfe111|3600601601282300044f94e5b78cbe111|3600601601282300046e16a6178cbe111|3600601601282300046f94e5b78cbe111|3600601601282300048e16a6178cbe111|360060160128230004ae16a6178cbe111|360060160128230004ce16a6178cbe111|360060160128230004ee16a6178cbe111|360060160128230004fc5f0cf72cfe111|3600601601282300050e16a6178cbe111|3600601601282300051c5f0cf72cfe111|3600601601282300052e16a6178cbe111|3600601601282300053c5f0cf72cfe111|3600601601282300055c5f0cf72cfe111|3600601601282300057c5f0cf72cfe111|3600601601282300059c5f0cf72cfe111|360060160128230005bc5f0cf72cfe111|360060160128230005dc5f0cf72cfe111|360060160128230005fc5f0cf72cfe111|3600601601282300061c5f0cf72cfe111|360060160128230006264545578cbe111|3600601601282300063c5f0cf72cfe111|360060160128230006464545578cbe111|3600601601282300065066d6e78cbe111|360060160128230006664545578cbe111|3600601601282300067066d6e78cbe111|360060160128230006864545578cbe111|3600601601282300069066d6e78cbe111|360060160128230006a64545578cbe111|360060160128230006b066d6e78cbe111|360060160128230006c64545578cbe111|360060160128230006d066d6e78cbe111|360060160128230006e64545578cbe111|360060160128230006f066d6e78cbe111|360060160128230007064545578cbe111|3600601601282300071066d6e78cbe111|360060160128230007264545578cbe111|360060160128230007464545578cbe111|360060160128230007664545578cbe111|360060160128230009fda68dc72cfe111|36006016012823000a1da68dc72cfe111|36006016012823000c0f3098442cfe111|36006016012823000c2f3098442cfe111|36006016012823000c4f3098442cfe111|36006016012823000c6f3098442cfe111|36006016012823000c8f3098442cfe111|36006016012823000c9602c468bd5e111|36006016012823000caf3098442cfe111|36006016012823000ccf3098442cfe111|36006016012823000cef3098442cfe111|36006016012823000d0f3098442cfe111|36006016012823000d2f3098442cfe111|36006016012823000d4f3098442cfe111|36006016012823000fe0aae5a42cfe111%\\", \\"r%.*%\\" ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free dcaf3756-1e12-401f-a4b1-8dc943a7f34f' (cwd None)
..
..
..
..
Thread-9381::DEBUG::2012-08-07 15:29:17,701::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%36006016012823000020bae5a42cfe111|36006016012823000060bae5a42cfe111|36006016012823000080bae5a42cfe111|3600601601282300009b5096878cbe111|360060160128230000a0bae5a42cfe111|360060160128230000bb5096878cbe111|360060160128230000c0bae5a42cfe111|360060160128230000db5096878cbe111|360060160128230000e0bae5a42cfe111|360060160128230000fb5096878cbe111|36006016012823000100bae5a42cfe111|3600601601282300011b5096878cbe111|36006016012823000120bae5a42cfe111|360060160128230002265fed572cfe111|360060160128230002465fed572cfe111|3600601601282300024c4e57478cbe111|360060160128230002665fed572cfe111|3600601601282300026c4e57478cbe111|360060160128230002865fed572cfe111|3600601601282300028c4e57478cbe111|360060160128230002a65fed572cfe111|360060160128230002ac4e57478cbe111|360060160128230002c65fed572cfe111|360060160128230002cc4e57478cbe111|360060160128230002e07326742cfe111|360060160128230002e65fed572cfe111|360060160128230002ec4e57478cbe111|360060160128230003065fed572cfe111|3600601601282300030c4e57478cbe111|360060160128230003265fed572cfe111|3600601601282300032c4e57478cbe111|3600601601282300032f94e5b78cbe111|360060160128230003465fed572cfe111|3600601601282300034c4e57478cbe111|3600601601282300034f94e5b78cbe111|360060160128230003665fed572cfe111|3600601601282300036f94e5b78cbe111|360060160128230003865fed572cfe111|3600601601282300038f94e5b78cbe111|360060160128230003af94e5b78cbe111|360060160128230003cf94e5b78cbe111|360060160128230003d4e306142cfe111|360060160128230003ef94e5b78cbe111|360060160128230003f4e306142cfe111|3600601601282300040f94e5b78cbe111|36006016012823000414e306142cfe111|3600601601282300042f94e5b78cbe111|36006016012823000434e306142cfe111|3600601601282300044f94e5b78cbe111|3600601601282300046e16a6178cbe111|3600601601282300046f94e5b78cbe111|3600601601282300048e16a6178cbe111|360060160128230004ae16a6178cbe111|360060160128230004ce16a6178cbe111|360060160128230004ee16a6178cbe111|360060160128230004fc5f0cf72cfe111|3600601601282300050e16a6178cbe111|3600601601282300051c5f0cf72cfe111|3600601601282300052e16a6178cbe111|3600601601282300053c5f0cf72cfe111|3600601601282300055c5f0cf72cfe111|3600601601282300057c5f0cf72cfe111|3600601601282300059c5f0cf72cfe111|360060160128230005bc5f0cf72cfe111|360060160128230005dc5f0cf72cfe111|360060160128230005fc5f0cf72cfe111|3600601601282300061c5f0cf72cfe111|360060160128230006264545578cbe111|3600601601282300063c5f0cf72cfe111|360060160128230006464545578cbe111|3600601601282300065066d6e78cbe111|360060160128230006664545578cbe111|3600601601282300067066d6e78cbe111|360060160128230006864545578cbe111|3600601601282300069066d6e78cbe111|360060160128230006a64545578cbe111|360060160128230006b066d6e78cbe111|360060160128230006c64545578cbe111|360060160128230006d066d6e78cbe111|360060160128230006e64545578cbe111|360060160128230006f066d6e78cbe111|360060160128230007064545578cbe111|3600601601282300071066d6e78cbe111|360060160128230007264545578cbe111|360060160128230007464545578cbe111|360060160128230007664545578cbe111|360060160128230009fda68dc72cfe111|36006016012823000a1da68dc72cfe111|36006016012823000c0f3098442cfe111|36006016012823000c2f3098442cfe111|36006016012823000c4f3098442cfe111|36006016012823000c6f3098442cfe111|36006016012823000c8f3098442cfe111|36006016012823000c9602c468bd5e111|36006016012823000caf3098442cfe111|36006016012823000ccf3098442cfe111|36006016012823000cef3098442cfe111|36006016012823000d0f3098442cfe111|36006016012823000d2f3098442cfe111|36006016012823000d4f3098442cfe111|36006016012823000fe0aae5a42cfe111%\\", \\"r%.*%\\" ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free dcaf3756-1e12-401f-a4b1-8dc943a7f34f' (cwd None)
Thread-9381::DEBUG::2012-08-07 15:29:20,796::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0
Comment 4 Eduardo Warszawski 2012-08-28 13:55:05 UTC 
http://gerrit.ovirt.org/#/c/7163/
Comment 5 Ayal Baron 2012-09-02 14:29:03 UTC 
Fede, http://gerrit.ovirt.org/#/c/7294 ?
Comment 6 Rami Vaknin 2012-09-03 12:54:12 UTC 
Could you please tell how many vgs should be after the above patch? will it be constant or still depends on the number of storage domains?
Comment 8 Ayal Baron 2012-09-30 21:18:24 UTC 
(In reply to comment #6)
> Could you please tell how many vgs should be after the above patch? will it
> be constant or still depends on the number of storage domains?

exactly the same number, it's just not in connectStoragePool, it's asynchronously in the domain monitoring threads.

dealing with number of lvm commands will be done in a separate set of patches.
Comment 9 Leonid Natapov 2012-10-14 13:37:53 UTC 
vdsm-4.9.6-36.0.el6_3.x86_64. Tested with 30 SD and 3 hosts. ConnectStoragePool was very quick. less than 30 seconds.
Comment 13 errata-xmlrpc 2012-12-04 19:04:58 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-1508.html
Note You need to log in before you can comment on or make changes to this bug.